DSBulk无法连接远程Cassandra集群加载CSV数据求助

核心问题 #

使用DSBulk将本地CSV加载至远程Cassandra集群时，出现两类问题：

1. 配置文件未被加载，默认连接127.0.0.1导致失败；
2. 命令行指定集群参数后，仍因连接被远程强制关闭而失败，但相同SSL参数可在IntelliJ正常连接集群。

报错信息 #

配置未加载时的报错 #

[driver] Error connecting to Node(endPoint=/127.0.0.1:9042, hostId=null, hashCode=2c61adb4)

连接集群失败的核心报错 #

Could not reach any contact point, make sure you've provided valid addresses

Caused by: An existing connection was forcibly closed by the remote host.

命令行指定参数后的详细报错 #

[driver] Error connecting to Node(endPoint=/169.XX.XXX.XX:9042, hostId=null, hashCode=2a38b2fe),
Suppressed: [driver|control|id: 0x17d0139b, L:/172.31.50.184:59702 - R:/169.XX.XXX.XXX:9042] Protocol initialization request, step 1 (OPTIONS): unexpected failure (com.datastax.oss.driver.api.core.connection.ClosedConnectionException: Unexpected error on channel).
     Caused by: Unexpected error on channel.
       Caused by: An existing connection was forcibly closed by the remote host.

认证推断信息 #

Username and password provided but auth provider not specified, inferring PlainTextAuthProvider

现有配置与执行命令 #

配置文件application.conf（存在语法错误） #

dsbulk {
  --dsbulk.connector.name = csv
  --dsbulk.connector.csv.url = <CSV_Path>
  --dsbulk.connector.csv.header true
  --datastax-java-driver.basic.contact-points = [ "169.XX.XXX.XX", "169.XX.XXX.XX", "169.XX.XXX.XX" ]
  --datastax-java-driver.advanced.auth-provider.username = <user_name>
  --datastax-java-driver.advanced.auth-provider.password = <pwd
  --dsbulk.schema.keyspace = <key space
  --dsbulk.schema.table = <table
  --datastax-java-driver.advanced.ssl-engine-factory.truststore-path = <cacerts path<br/>
  --datastax-java-driver.advanced.ssl-engine-factory.truststore-password = <pwd
  --datastax-java-driver.advanced.resolve-contact-points = true
}

执行命令1（未加载配置） #

$ dsbulk load -url CSV Path**

执行命令2（指定参数但未加SSL配置） #

$ dsbulk load -url CSV Path -k keysapce -t table -h "[ "169.XX.XXX.XX", "169.XX.XXX.XX", "169.XX.XXX.XX" ]" -u userName -p pwd

排查建议 #

• 修复配置文件语法错误：DSBulk配置文件中不需要给配置项加--前缀，同时修正未闭合的变量占位符（如<pwd需改为<pwd>）和错误的路径格式（去掉<br/>），修正后示例：

  dsbulk {
    connector.name = csv
    connector.csv.url = "<CSV_Path>"
    connector.csv.header = true
    datastax-java-driver.basic.contact-points = [ "169.XX.XXX.XX", "169.XX.XXX.XX", "169.XX.XXX.XX" ]
    datastax-java-driver.advanced.auth-provider.username = "<user_name>"
    datastax-java-driver.advanced.auth-provider.password = "<pwd>"
    schema.keyspace = "<keyspace>"
    schema.table = "<table>"
    datastax-java-driver.advanced.ssl-engine-factory.truststore-path = "<cacerts_path>"
    datastax-java-driver.advanced.ssl-engine-factory.truststore-password = "<pwd>"
    datastax-java-driver.advanced.resolve-contact-points = true
  }
  

• 确保配置文件被加载：将修正后的application.conf放在执行DSBulk命令的当前目录，或通过--conf-file /full/path/to/application.conf参数指定配置文件路径；也可检查环境变量DSBULK_CONF是否指向正确的配置目录。
• 修正命令行参数：
  • -h参数无需嵌套引号和方括号，直接用逗号分隔节点：-h 169.XX.XXX.XX,169.XX.XXX.XX,169.XX.XXX.XX
  • 修正keysapce的拼写错误为keyspace
  • 补充SSL相关参数，因为IntelliJ连接用了SSL，命令行需显式指定：

    $ dsbulk load -url "CSV Path" -k keyspace -t table -h "169.XX.XXX.XX,169.XX.XXX.XX,169.XX.XXX.XX" -u userName -p pwd --datastax-java-driver.advanced.ssl-engine-factory.truststore-path "<cacerts_path>" --datastax-java-driver.advanced.ssl-engine-factory.truststore-password "<pwd>"
    

• 网络端口验证：确认本地机器能访问Cassandra节点的9042端口，可通过telnet 169.XX.XXX.XX 9042或nc -zv 169.XX.XXX.XX 9042测试连通性。

替代方案（Sybase数据归档到Cassandra） #

• Java程序集成驱动：直接使用DataStax Java Driver结合Sybase JDBC驱动，批量读取Sybase数据后分批次写入Cassandra，适合需要自定义业务逻辑的场景。
• ETL工具Apache NiFi：配置Sybase数据源处理器和Cassandra目标处理器，可视化设置定时归档任务，无需编写大量代码。
• DSBulk Java API调用：在Java项目中引入DSBulk的依赖，通过API调用加载逻辑，灵活控制加载的启动、暂停、日志等流程。

内容的提问来源于stack exchange，提问作者harish bollina