在GKE上用sstableloader向Scylla DB加载数据遇Netty通道创建错误

问题现象 #

执行sstableloader加载数据时出现连接超时，后续触发空指针异常，GKE环境下也复现相同问题：

root@scylla-chronicle-s-1:~# sstableloader -d 10.110.68.9 /var/lib/scylla/data/connections/by_date-4ce6c340f5dd11e980df000000000002/snapshots/1658173631835

Using /etc/scylla/scylla.yaml as the config file
===== Using optimized driver!!! =====
WARN  19:51:22,937 Error creating netty channel to /10.110.117.172:9042
com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /10.110.117.172:9042
        at com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.shaded.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.shaded.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]
ERROR 19:51:22,942 Unexpected error while executing task
java.lang.NullPointerException: null
        at com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:838) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.driver.core.SessionManager.removePool(SessionManager.java:437) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.driver.core.SessionManager.onDown(SessionManager.java:525) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.driver.core.Cluster$Manager.onDown(Cluster.java:2033) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.driver.core.Cluster$Manager.access$1200(Cluster.java:1393) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.driver.core.Cluster$Manager$5.runMayThrow(Cluster.java:1988) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_292]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
        at com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]
WARN  19:51:22,948 Error creating pool to /10.110.117.172:9042

排查与解决步骤 #

1. 确认网络连通性 #

• 从运行sstableloader的节点测试目标节点9042端口：

  nc -zv 10.110.117.172 9042
  # 或使用telnet
  telnet 10.110.117.172 9042
  

• GKE环境额外检查：集群网络策略是否允许Pod间9042端口通信；ScyllaDB服务的ClusterIP/NodePort是否可正常访问；云厂商防火墙是否开放该端口。

2. 验证ScyllaDB节点状态 #

• 登录目标节点确认服务运行状态：

  systemctl status scylla-server
  

• 检查9042端口是否被监听：

  ss -tulpn | grep 9042
  

• 查看ScyllaDB日志排查启动或端口绑定错误：

  journalctl -u scylla-server -f
  

3. 调整sstableloader参数 #

• 增加连接超时时间，避免网络延迟导致超时：

  sstableloader --connect-timeout 30000 -d 10.110.68.9 /var/lib/scylla/data/connections/by_date-4ce6c340f5dd11e980df000000000002/snapshots/1658173631835
  

• 指定多个集群节点IP，降低单点依赖：

  sstableloader -d 10.110.68.9,10.110.117.172 <sstable-path>
  

• 检查/etc/scylla/scylla.yaml中的rpc_address配置，确保绑定的是可被外部访问的IP（不要设为localhost）。

4. GKE环境专属检查 #

• 确认ScyllaDB Pod的livenessProbe和readinessProbe正常，Pod处于Running/Ready状态。
• 检查StatefulSet的Headless Service配置，确保DNS解析正常。
• 在集群内临时启动测试Pod执行sstableloader，排除外部网络问题。

5. 空指针异常处理 #

• 空指针是连接超时后的次生错误，解决网络/节点连通问题后通常会自动消失。
• 如果问题持续，升级sstableloader到对应ScyllaDB版本的最新版，该NPE可能是已知bug已被修复。

内容的提问来源于stack exchange，提问作者Aditya Bibave