You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

为何Spark Cassandra Connector的LocalNodeFirstLoadBalancingPolicy会添加非本地DC节点?

LocalNodeFirstLoadBalancingPolicy 跨DC节点被异常添加问题说明

根据LocalNodeFirstLoadBalancingPolicy的官方文档说明:

优先选择本地节点,之后以随机顺序选择本地数据中心(DC)的节点,绝不会选择其他数据中心的节点。对于写入操作,如果语句设置了路由键,该负载均衡策略(LBP)具备令牌感知能力——相比其他节点,会优先选择计算出的令牌对应的副本节点。

但在实际Spark作业运行日志中,观察到集群内所有节点都被程序识别纳入,具体日志片段:

21/05/05 10:08:40 INFO CassandraWriter$: Setting local_dc: DC1
21/05/05 10:08:40 INFO CassandraWriter$: Writing to DC: DC1, available host ips: x.x.x.54,x.x.x.237,x.x.x.168,x.x.x.197,x.x.x.219
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.219:9042 added
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.237:9042 added
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.54:9042 added
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.238:9042 added
21/05/05 10:08:41 INFO LocalNodeFirstLoadBalancingPolicy: Added host x.x.x.238 (DC2)
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.168:9042 added
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.42:9042 added
21/05/05 10:08:41 INFO LocalNodeFirstLoadBalancingPolicy: Added host x.x.x.42 (DC2)
21/05/05 10:08:41 INFO Cluster: New Cassandra host /x.x.x.109:9042 added
21/05/05 10:08:41 INFO LocalNodeFirstLoadBalancingPolicy: Added host x.x.x.109 (DC2)

核心疑问:为什么DC2的节点会被策略添加?按照对策略的理解,协调节点应当始终从配置的local_dc范围内选取。
测试过程中尝试过不设置spark.cassandra.connection.local_dc参数运行数据摄入任务,也观测到了完全一致的日志表现。
对应的写入实现代码如下:

records.write.cassandraFormat(table, keySpace)
  .mode(SaveMode.Append)
  .option(CassandraConnectorConf.LocalDCParam.name, cassandraDC.name)
  .option(CassandraConnectorConf.ConnectionHostParam.name, cassandraDC.availableHosts.mkString(","))
  .save()

补充场景说明:Spark集群与Cassandra集群部署相互独立,当前使用场景为从Spark集群向Cassandra集群批量写入数据。


内容的提问来源于stack exchange,提问作者Ana

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级,最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分), 限时 9.9 元起

最近更新时间:2026.10.22 23:45:05