Scylla节点出现PT2S超时错误，如何排查性能瓶颈？

问题背景 #

使用Java驱动操作单节点Scylla时，出现间歇性读写超时错误，错误信息为Query timed out after PT2S。已排除以下因素：

• 单节点部署，服务器无负载（报错时也无负载）
• 统计数据正常，压缩活动几乎为零
• 无大分区告警或相关统计

不想通过提高超时时间解决，需找出节点响应缓慢的根本原因，还可从以下方面排查：

上下文信息 #

• 硬件：通用硬件，4核CPU，8GB内存
• 服务器状态：无其他主要进程运行
• 键空间与表结构：仅包含uzzstore.chunks表，结构如下：

CREATE TABLE uzzstore.chunks (
    id text PRIMARY KEY,
    size bigint
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.0
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

• cfstats统计信息：

$ nodetool flush uzzstore chunks;
$ nodetool cfstats uzzstore.chunks;
Total number of tables: 69
----------------
Keyspace : uzzstore
    Read Count: 13347248
    Read Latency: 0.004633434922315072 ms
    Write Count: 46506776
    Write Latency: 1.3758554237343822E-5 ms
    Pending Flushes: 0
        Table: chunks
        SSTable count: 15
        SSTables in each level: [15/4]
        Space used (live): 17960566784
        Space used (total): 17960566784
        Space used by snapshots (total): 0
        Off heap memory used (total): 164689156
        SSTable Compression Ratio: 0.827822
        Number of partitions (estimate): 117347789
        Memtable cell count: 0
        Memtable data size: 0
        Memtable off heap memory used: 0
        Memtable switch count: 178
        Local read count: 13347248
        Local read latency: 4.633 ms
        Local write count: 46506776
        Local write latency: 0.014 ms
        Pending flushes: 0
        Percent repaired: 0.0
        Bloom filter false positives: 59418
        Bloom filter false ratio: 0.02055
        Bloom filter space used: 155733448
        Bloom filter off heap memory used: 156599748
        Index summary off heap memory used: 8089408
        Compression metadata off heap memory used: 0
        Compacted partition minimum bytes: 73
        Compacted partition maximum bytes: 103
        Compacted partition mean bytes: 103
        Average live cells per slice (last five minutes): 0.0
        Maximum live cells per slice (last five minutes): 0
        Average tombstones per slice (last five minutes): 0.0
        Maximum tombstones per slice (last five minutes): 0
        Dropped Mutations: 0

• 大分区查询结果：

cqlsh> SELECT * FROM system.large_partitions;

 keyspace_name | table_name | sstable_name | partition_size | partition_key | compaction_time | rows
---------------+------------+--------------+----------------+---------------+-----------------+------

(0 rows)

• 写入错误（执行INSERT INTO "chunks" ("id" ,"size" ) VALUES (?,?)时）：

Caused by: java.sql.SQLTransientException: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
        at com.ing.data.cassandra.jdbc.CassandraPreparedStatement.doExecute(CassandraPreparedStatement.java:216)
        at com.ing.data.cassandra.jdbc.CassandraPreparedStatement.executeUpdate(CassandraPreparedStatement.java:308)
        at com.j256.ormlite.jdbc.JdbcDatabaseConnection.insert(JdbcDatabaseConnection.java:185)
        at com.j256.ormlite.stmt.mapped.MappedCreate.insert(MappedCreate.java:91)
        ... 18 more
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
        at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:34)
        at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
        at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
        at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
        at com.ing.data.cassandra.jdbc.CassandraPreparedStatement.doExecute(CassandraPreparedStatement.java:214)
        ... 21 more

• 读取错误（执行包含46个ID的SELECT "id" FROM "chunks" WHERE "id" IN (?,...,?)时）：

Caused by: java.sql.SQLTransientException: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
        at com.ing.data.cassandra.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:136)
        at com.ing.data.cassandra.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:537)
        at com.ing.data.cassandra.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:529)
        at com.ing.data.cassandra.jdbc.CassandraConnection.prepareStatement(CassandraConnection.java:89)
        at com.j256.ormlite.jdbc.JdbcDatabaseConnection.compileStatement(JdbcDatabaseConnection.java:143)
        at com.j256.ormlite.stmt.mapped.MappedPreparedStmt.compile(MappedPreparedStmt.java:57)
        at com.j256.ormlite.stmt.StatementExecutor.buildIterator(StatementExecutor.java:252)
        at com.j256.ormlite.stmt.StatementExecutor.query(StatementExecutor.java:202)
        at com.j256.ormlite.dao.BaseDaoImpl.query(BaseDaoImpl.java:286)
        at com.j256.ormlite.stmt.QueryBuilder.query(QueryBuilder.java:384)
        at com.j256.ormlite.stmt.Where.query(Where.java:519)
        at ste.uzz.store.DBChunkStore.contains(DBChunkStore.java:126)
        ... 9 more
Caused by: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
        at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:34)
        at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
        at com.datastax.oss.driver.internal.core.cql.CqlPrepareSyncProcessor.process(CqlPrepareSyncProcessor.java:59)
        at com.datastax.oss.driver.internal.core.cql.CqlPrepareSyncProcessor.process(CqlPrepareSyncProcessor.java:31)
        at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
        at com.datastax.oss.driver.api.core.cql.SyncCqlSession.prepare(SyncCqlSession.java:224)
        at com.ing.data.cassandra.jdbc.CassandraPreparedStatement.<init>(CassandraPreparedStatement.java:131)
        ... 20 more

排查方向 #

1. Scylla内部线程与调度问题 #

• 检查Scylla的system.log，重点查找GC停顿、线程阻塞或调度延迟相关日志。即使服务器整体负载低，Scylla内部的某个线程池（如IO线程、协调器线程）可能出现瞬时阻塞。
• 使用nodetool tpstats查看线程池状态，关注Pending请求数、Blocked线程数是否有异常波动。
• 检查Scylla的内存配置：8GB内存下，确认memory.yaml中total_memory_in_mb设置是否合理，是否存在内存分配不足导致的频繁内存回收。

2. 磁盘IO性能波动 #

• 即使整体负载低，磁盘可能存在瞬时IO阻塞（如磁盘缓存刷新、底层存储系统的延迟）。使用iostat -x 1实时监控磁盘的%util、await、svctm指标，看是否有瞬时的高延迟。
• 检查磁盘是否有硬件问题：查看dmesg或smartctl输出，确认是否存在磁盘坏道、IO错误。
• 确认Scylla的数据目录是否在性能稳定的存储介质上（如避免使用共享存储或性能波动的云盘）。

3. Java驱动与连接配置问题 #

• 检查驱动的连接池配置：是否存在连接池耗尽、连接超时设置不合理的情况。比如advanced.connection.pool.size是否与服务器资源匹配，是否有连接泄漏。
• 查看驱动的speculative_retry设置：当前表的speculative_retry是99.0PERCENTILE，单节点下重试机制可能无法生效，反而增加负载，需确认驱动的重试逻辑是否有异常。
• 检查是否存在准备语句缓存问题：第二个错误是在prepare语句时超时，可能是驱动的准备语句缓存未正确配置，导致频繁重复准备语句，增加节点负担。

4. Scylla的Compaction与后台任务 #

• 虽然cfstats显示压缩活动几乎为零，但SizeTieredCompactionStrategy可能存在瞬时的compaction触发（比如当SSTable数量达到阈值时）。查看system.log中是否有Starting compaction for相关日志，确认是否在超时发生时刚好有compaction启动。
• 检查是否有其他后台任务：如快照清理、索引维护等，这些任务可能瞬时占用资源。

5. 网络与TCP配置问题 #

• 即使是本地连接（驱动和Scylla在同一机器），TCP层的配置也可能导致问题：比如tcp_tw_reuse、tcp_timestamps等参数是否优化，是否存在TCP连接的瞬时阻塞。
• 如果是远程连接，检查网络链路是否有瞬时丢包或延迟，使用ping、mtr工具持续监控。

6. 表结构与查询优化 #

• 对于IN查询：46个ID的IN查询需分别查找每个分区，若这些分区分布在不同的SSTable中，可能导致瞬时IO放大。可尝试将IN查询拆分成多个单分区查询，看是否还会超时。
• 检查caching配置：当前表的caching是{'keys': 'ALL', 'rows_per_partition': 'ALL'}，确认缓存是否正常工作，是否存在缓存失效后大量冷数据读取导致的延迟。

内容的提问来源于stack exchange，提问作者Ste