生产环境Cassandra插入偶发TimeoutError，无WriteTimeoutError

问题现象 #

• 3节点生产环境，采用LOCAL_QUORUM一致性级别执行插入操作时，偶发失败
• 仅抛出Cassandra::Errors::TimeoutError，未出现Cassandra::Errors::WriteTimeoutError或Cassandra::Errors::NoHostsAvailable错误
• Cassandra集群日志无异常，仅应用端日志记录该错误
• 每日约发生1000次，调用方重试后通常可成功，怀疑Ruby Cassandra驱动存在问题

环境配置 #

Ruby与驱动版本 #

ruby '~> 2.7'
gem "cassandra-driver", "~> 3.2.5"

一致性与负载均衡策略 #

consistency:           :local_quorum,

load_balancing_policies = {
    dc_aware_round_robin: Cassandra::LoadBalancing::Policies::DCAwareRoundRobin.new(
        datacenter,
        cassandra_used_hosts_per_remote_dc
    ),
    round_robin: Cassandra::LoadBalancing::Policies::RoundRobin.new
}

Cassandra连接配置参数 #

CASSANDRA_CONNECT_TIMEOUT_MS: '600'
CASSANDRA_CONSISTENCY: LOCAL_QUORUM
CASSANDRA_RECONNECT_INITIAL_INTERVAL_MS: '100'
CASSANDRA_RECONNECT_MAX_INTERVAL_MS: '3000'
CASSANDRA_RECONNECT_MAX_RETRIES: '5'
CASSANDRA_RETRIES: '5'
CASSANDRA_RETRY_MAX_MS: '3000'
CASSANDRA_RETRY_MIN_MS: '100'

驱动核心代码片段 #

查看lib/cassandra/future.rb中的get方法：

# 返回Future的结果或抛出错误
#
# @note 该方法会阻塞线程直到Future完成或超时
#
# @param timeout [nil, Numeric] 等待Future完成的最大秒数，传入nil则无限等待
#
# @raise [Errors::TimeoutError] 等待时间超过超时阈值时抛出
# @raise [Exception] Future执行失败时抛出原始异常
#
# @return [Object] Future完成后返回的值
def get(timeout = nil)
  @signal.get(timeout)
end

错误详情 #

错误信息 #

Cassandra::Errors::TimeoutError
Timed out

错误栈追踪 #

/srv/_versions/events/events-202304261636-9ba0b992cd-master/vendor/bundle/ruby/2.7.0/gems/cassandra-driver-3.2.5/lib/cassandra/future.rb:637:in 'get',
/srv/_versions/events/events-202304261636-9ba0b992cd-master/vendor/bundle/ruby/2.7.0/gems/cassandra-driver-3.2.5/lib/cassandra/future.rb:402:in 'get',
/srv/_versions/events/events-202304261636-9ba0b992cd-master/vendor/bundle/ruby/2.7.0/gems/cassandra-driver-3.2.5/lib/cassandra/session.rb:127:in 'execute'

内容的提问来源于stack exchange，提问作者AVI