Spark Submit配置解决HBase正则扫描SocketTimeoutException问题

Absolutely! You don’t need to modify code or cluster config files—you can directly inject HBase configuration properties through spark-submit using the --conf flag. Here’s exactly how to do it:

When submitting your Spark job, add the following --conf arguments to override the timeout settings:

spark-submit \
  --class com.your.package.YourMainClass \
  --conf "spark.hadoop.hbase.rpc.timeout=1800000" \
  --conf "spark.hadoop.hbase.client.scanner.timeout.period=1800000" \
  --conf "spark.hadoop.hbase.client.operation.timeout=1800000" \
  your-spark-application.jar

How this works: #

Spark automatically propagates configuration properties prefixed with spark.hadoop. to the underlying Hadoop Configuration instance that HBase uses. This means your HBase client will pick up these extended timeout values without any code changes or cluster config edits.

You can use this pattern for any other HBase-related configs you need to adjust later—just prepend spark.hadoop. to the original HBase config key.

内容的提问来源于stack exchange，提问作者Ignacio Alorre