Tomcat 9使用UnixDomainSocket连接器时触发服务崩溃问题

故障现象 #

• 基于Tomcat 9新增的Unix Domain Socket特性配置服务时，无公开可运行的参考配置，使用curl向socket发起请求，第二次请求就会触发服务完全卡死崩溃
• 复现使用的curl命令：

sudo curl  -kv -H "Connection: close" -H "x-forwarded-for: 127.0.0.1" http://localhost/hawtio/auth/login --unix-socket /opt/app-name/http.sock

运行环境 #

• Ubuntu 18.04
• Tomcat 9.0.64
• 最新版Java 11
• Apache Portable Runtime / Tomcat Native 1.2.35

故障日志 #

首次请求触发空指针异常，堆栈如下：

#first request
SEVERE [http-apr-/opt/app-name/http.sock-exec-1] org.apache.coyote.http11.Http11Processor.service Error processing request
	java.lang.NullPointerException
		at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1770)
		at java.base/java.util.regex.Matcher.reset(Matcher.java:416)
		at java.base/java.util.regex.Matcher.<init>(Matcher.java:253)
		at java.base/java.util.regex.Pattern.matcher(Pattern.java:1133)
		at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:624)
		at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
		at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
		at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
		at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
		at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
		at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2113)
		at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
		at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.base/java.lang.Thread.run(Thread.java:829)

第二次请求触发Acceptor线程空指针，服务彻底卡死，堆栈如下：

# second request
SEVERE [http-apr-/opt/app-name/http.sock-Acceptor] jdk.internal.reflect.NativeMethodAccessorImpl.invoke Error allocating socket processor
	java.lang.NullPointerException
		at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:820)
		at org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(AprEndpoint.java:87)
		at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:149)
		at java.base/java.lang.Thread.run(Thread.java:829)

故障复现配置 #

核心server.xml配置片段：

<Service name="CatalinaLocal">
    <Connector
        protocol="org.apache.coyote.http11.Http11AprProtocol"
        unixDomainSocketPath="/opt/app-name/http.sock" />
    <Engine
        defaultHost="localhost"
        name="Catalina">
        <Host
            name="localhost"
            appBase="webapps"
            unpackWARs="true"
            autoDeploy="false"
            deployIgnore="(?!.*hawtio).*">
            <Valve
                className="org.apache.catalina.valves.RemoteIpValve" />
        </Host>
    </Engine>
</Service>

• 移除配置中的RemoteIpValve后，崩溃现象完全一致，说明故障和该Valve无直接关联。

根因定位 #

故障核心原因是Tomcat APR连接器的多处逻辑未适配Unix Domain Socket（UDS）场景：Unix Domain Socket本身是本机进程间通信机制，不存在TCP协议特有的远端IP、远端端口属性，相关属性调用会返回null，直接触发空指针：

1. 首次请求时，RemoteIpValve逻辑会读取请求远端地址做正则匹配，拿到null值直接触发第一次空指针
2. 第二次请求进入连接Accept流程时，AprEndpoint类内置的重复accept检测逻辑未做null判断，直接读取远端端口、远端地址做比较，触发空指针直接打崩Acceptor线程，导致服务完全卡死

该重复accept检测逻辑仅存在于APR（基于JNI调用Apache Portable Runtime）连接器中，设计时仅考虑了TCP套接字场景，未覆盖UDS场景，属于Tomcat的代码适配疏漏，对应问题逻辑片段如下：

// Do the duplicate accept check here rather than in serverSocketaccept()
// so we can cache the results in the SocketWrapper
AprSocketWrapper wrapper = new AprSocketWrapper(socket, this);
// Bug does not affect Windows. Skip the check on that platform.
if (!JrePlatform.IS_WINDOWS) {
    long currentNanoTime = System.nanoTime();
    if (wrapper.getRemotePort() == previousAcceptedPort) {
        if (wrapper.getRemoteAddr().equals(previousAcceptedAddress)) {
            if (currentNanoTime - previousAcceptedSocketNanoTime < 1000) {
                throw new IOException(sm.getString("endpoint.err.duplicateAccept"));
            }
        }
    }
    previousAcceptedPort = wrapper.getRemotePort();
    previousAcceptedAddress = wrapper.getRemoteAddr();
    previousAcceptedSocketNanoTime = currentNanoTime;
}

解决方案 #

1. 临时规避方案：不要使用APR协议连接器绑定Unix Domain Socket，替换为NIO连接器即可正常使用，修改Connector配置如下：

   <Connector
       protocol="org.apache.coyote.http11.Http11NioProtocol"
       unixDomainSocketPath="/opt/app-name/http.sock" />
   

2. 永久修复方案：若必须使用APR连接器，可升级Tomcat版本到9.0.65及以上，该版本已修复Unix Domain Socket场景下的空指针问题，相关逻辑增加了UDS场景判断，会跳过TCP专属的重复accept检测，同时给远端地址、端口属性设置默认值，避免空指针。

内容的提问来源于stack exchange，提问作者Jonathan S. Fisher