InetAddress.getByName引发K8s容器应用启动死锁求助
应用启动时因InetAddress相关死锁挂起的排查与修复
问题描述
应用启动时会出现挂起,根源是和InetAddress.getByName相关的死锁场景,但不清楚修复方案。涉及的两个线程都不受直接控制:
- 处于BLOCKED状态的线程正在启动Prometheus HTTP服务器,对应代码:
new InetSocketAddress("0.0.0.0", somePort) - 处于RUNNABLE状态的线程和ZIO Http客户端库相关,调用Netty客户端逻辑,对应代码:
static final InetAddress INET6_ANY = InetAddress.getByName("::") static final InetAddress INET_ANY = InetAddress.getByName("0.0.0.0")
已知InetAddress调用可能阻塞,但针对本地地址0.0.0.0为何会永久挂起?应用运行在Kubernetes容器中,容器内/etc/hosts内容如下:
$ cat /etc/hosts # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.42.126.184 my-app-756f44d67-tgr5b
注:问题并非每次都能复现,但近期已出现多次。请问这是库的误用问题,还是Kubernetes环境中遗漏了必要配置?
线程Dump详情
BLOCKED状态线程
"ZScheduler-Worker-6" #30 daemon prio=5 os_prio=0 cpu=276.23ms elapsed=7541.20s tid=0x00007f7bd54d4880 nid=0x55 waiting for monitor entry [0x00007f7b663f4000] java.lang.Thread.State: BLOCKED (on object monitor) at jdk.internal.loader.NativeLibraries.loadLibrary(java.base@17.0.10/Unknown Source) - waiting to lock <0x00000000a02e9a90> (a java.util.HashSet) at jdk.internal.loader.NativeLibraries.loadLibrary(java.base@17.0.10/Unknown Source) at jdk.internal.loader.NativeLibraries.findFromPaths(java.base@17.0.10/Unknown Source) at jdk.internal.loader.NativeLibraries.loadLibrary(java.base@17.0.10/Unknown Source) at jdk.internal.loader.NativeLibraries.loadLibrary(java.base@17.0.10/Unknown Source) at jdk.internal.loader.BootLoader.loadLibrary(java.base@17.0.10/Unknown Source) at java.net.InetAddress.<clinit>(java.base@17.0.10/Unknown Source) at java.net.InetSocketAddress.<init>(java.base@17.0.10/Unknown Source) at io.prometheus.metrics.exporter.httpserver.HTTPServer$Builder.makeInetSocketAddress(HTTPServer.java:209) at io.prometheus.metrics.exporter.httpserver.HTTPServer$Builder.buildAndStart(HTTPServer.java:197) at io.opentelemetry.exporter.prometheus.PrometheusHttpServer.<init>(PrometheusHttpServer.java:71) at io.opentelemetry.exporter.prometheus.PrometheusHttpServerBuilder.build(PrometheusHttpServerBuilder.java:68) at com.myapp.metrics.sdk.PrometheusMetricReader$.$anonfun$startReader$2(PrometheusMetricReader.scala:21) at com.myapp.metrics.sdk.PrometheusMetricReader$$$Lambda$1109/0x00007f7b7840e078.apply(Unknown Source) at zio.ZIOCompanionVersionSpecific.$anonfun$attempt$1(ZIOCompanionVersionSpecific.scala:100) at zio.ZIOCompanionVersionSpecific$$Lambda$430/0x00007f7b782ba000.apply(Unknown Source) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:904) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381) at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504) at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220) at zio.internal.FiberRuntime.run(FiberRuntime.scala:139) at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:478)
锁定状态线程
"ZScheduler-Worker-20" #44 daemon prio=5 os_prio=0 cpu=191.27ms elapsed=7541.20s tid=0x00007f7bd54e2e70 nid=0x63 in Object.wait() [0x00007f7b655e3000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.LinuxSocket.unsafeInetAddrByName(LinuxSocket.java:364) - waiting on the Class initialization monitor for java.net.InetAddress at io.netty.channel.epoll.LinuxSocket.<clinit>(LinuxSocket.java:42) at jdk.internal.loader.NativeLibraries.load(java.base@17.0.10/Native Method) at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(java.base@17.0.10/Unknown Source) at jdk.internal.loader.NativeLibraries.loadLibrary(java.base@17.0.10/Unknown Source) - locked <0x00000000a02e9a90> (a java.util.HashSet) at jdk.internal.loader.NativeLibraries.loadLibrary(java.base@17.0.10/Unknown Source) at java.lang.ClassLoader.loadLibrary(java.base@17.0.10/Unknown Source) at java.lang.Runtime.load0(java.base@17.0.10/Unknown Source) at java.lang.System.load(java.base@17.0.10/Unknown Source) at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.10/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.10/Unknown Source) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.10/Unknown Source) at java.lang.reflect.Method.invoke(java.base@17.0.10/Unknown Source) at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:430) at java.security.AccessController.executePrivileged(java.base@17.0.10/Unknown Source) at java.security.AccessController.doPrivileged(java.base@17.0.10/Unknown Source) at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:422) at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:388) at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:218) at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:323) at io.netty.channel.epoll.Native.<clinit>(Native.java:85) at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:40) at zio.http.netty.ChannelFactories$Client$.$anonfun$fromConfig$4(ChannelFactories.scala:83) at zio.http.netty.ChannelFactories$Client$$$Lambda$966/0x00007f7b783d3e60.apply(Unknown Source) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967) at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967) at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381) at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504) at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220) at zio.internal.FiberRuntime.run(FiberRuntime.scala:139) at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:478)
死锁原因分析
这是典型的类初始化死锁:
- Netty/ZIO客户端线程(线程20):
- 正在加载Netty的
LinuxSocket类,该类静态初始化块中调用unsafeInetAddrByName,需要等待InetAddress类完成初始化。 - 同时已持有
NativeLibraries中的HashSet锁(地址<0x00000000a02e9a90>)。
- 正在加载Netty的
- Prometheus服务器线程(线程6):
- 创建
InetSocketAddress时触发InetAddress类的静态初始化,初始化过程需要加载底层本地库,因此尝试获取NativeLibraries的HashSet锁。 - 但该锁已被线程20持有,线程6进入BLOCKED状态。
- 创建
双方互相等待对方持有的资源:线程20等InetAddress初始化完成,线程6等NativeLibraries锁释放,形成死锁。
本地地址0.0.0.0触发问题的原因:InetAddress类的静态初始化是第一次使用该类时触发,不管解析本地还是远程地址,首次调用都会执行初始化逻辑,而初始化涉及本地库加载,就可能和其他线程的本地库加载操作产生锁竞争。问题偶发是因为只有当两个线程刚好在类初始化和本地库加载的交叉点同时执行时才会触发,属于时序性竞争问题。
修复方案
- 提前初始化InetAddress类:在应用启动最早期(比如main方法开头、多线程启动前),主动触发
InetAddress类初始化,例如调用InetAddress.getByName("0.0.0.0")或直接引用InetAddress.class。后续多线程调用时,InetAddress已完成初始化,避免触发类初始化逻辑。 - 提前加载Netty Native库:在应用启动阶段调用
Epoll.isAvailable(),提前完成LinuxSocket类的静态初始化,避免和InetAddress初始化产生竞争。 - 升级相关库版本:检查Netty、ZIO Http、Prometheus exporter的最新版本,看是否有修复类初始化死锁的补丁,比如Netty可能优化了
LinuxSocket的初始化逻辑,避免在静态块中依赖InetAddress初始化。
Kubernetes环境配置建议
该问题和Kubernetes的/etc/hosts配置无关,当前hosts文件是正常的Kubernetes默认配置。无需修改hosts或添加额外DNS配置,死锁根源是JVM类初始化和本地库加载的锁竞争,而非DNS解析问题。
内容的提问来源于stack exchange,提问作者Gaël J
相关产品推荐
相关产品推荐

