Java向Spark Submit传递长配置字符串致类加载失败的解决方法

通过Java应用向bash脚本传递完整Spark作业配置的长字符串参数，替代直接在spark-submit命令中配置参数时，出现类加载失败错误：

22/11/16 12:40:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: Failed to load class "com.at.es_parent_child.SegmentIcebergEsV2".

相关代码片段 #

Bash脚本片段 #

CONF1=$1
CONF='--class "com.at.es_parent_child.SegmentIcebergEsV2" --master yarn --deploy-mode client --queue llap --num-executors 3 --driver-memory 1024m --executor-memory 1024m --executor-cores 4 --name [564889711]es_parent_child.[0].1668574353481 --conf "spark.executor.extraClassPath=/etc/hbase/conf" --conf "spark.driver.extraClassPath=/etc/hbase/conf" --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.max.executor.failures=100" --conf "spark.rdd.compress=true" --conf "spark.sql.debug.maxToStringFields=2000" --conf "spark.sql.hive.convertMetastoreParquet=false" --conf "spark.default.parallelism=50" --conf "spark.debug.maxToStringFields=2000" --conf "hbase.defaults.for.version.skip=true" --conf "spark.yarn.executor.memoryOverhead=1024" --conf "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog" --conf "spark.sql.catalog.spark_catalog.type=hive" --conf "spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog" --conf "spark.sql.catalog.iceberg.type=hive" --conf "spark.sql.adaptive.coalescePartitions.enabled=true" --files "/etc/hbase/conf/hbase-site.xml,/usr/hdp/current/hive-client/conf/hive-site.xml"'
echo "--------------------------------------------"
echo $CONF
echo "--------------------------------------------"

sudo -u cdpcore /bin/sh /build/iceberg/spark-3.0.1-bin-hadoop2.7/bin/spark-submit $CONF --jars $(echo $JAR_LIB/*.jar | tr ' ' ',') $JAR_MAIN "$2" $3 $4 $5 &

# 直接执行echo输出的命令时Spark能正常运行
echo sudo -u cdpcore /bin/sh /build/iceberg/spark-3.0.1-bin-hadoop2.7/bin/spark-submit $CONF --jars $(echo $JAR_LIB/*.jar | tr ' ' ',') $JAR_MAIN "$2" $3 $4 $5 &

Java代码片段 #

String CONF = "--class \"com.at.es_parent_child.SegmentIcebergEsV2\" --master yarn --deploy-mode client --queue llap --num-executors 3 --driver-memory 1024m --executor-memory 1024m --executor-cores 4 --name [564889711]es_parent_child.[0].1668574353481 --conf \"spark.executor.extraClassPath=/etc/hbase/conf\" --conf \"spark.driver.extraClassPath=/etc/hbase/conf\" --conf \"spark.serializer=org.apache.spark.serializer.KryoSerializer\" --conf \"spark.max.executor.failures=100\" --conf \"spark.rdd.compress=true\" --conf \"spark.sql.debug.maxToStringFields=2000\" --conf \"spark.sql.hive.convertMetastoreParquet=false\" --conf \"spark.default.parallelism=50\" --conf \"spark.debug.maxToStringFields=2000\" --conf \"hbase.defaults.for.version.skip=true\" --conf \"spark.yarn.executor.memoryOverhead=1024\" --conf \"spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog\" --conf \"spark.sql.catalog.spark_catalog.type=hive\" --conf \"spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog\" --conf \"spark.sql.catalog.iceberg.type=hive\" --conf \"spark.sql.adaptive.coalescePartitions.enabled=true\" --files \"/etc/hbase/conf/hbase-site.xml,/usr/hdp/current/hive-client/conf/hive-site.xml\"";

String[] cmd = {"/bin/sh", System.getProperty("user.dir") + "/spark_job.sh", CONF,
                            zKUrl,""+ task.getPortalId(), task.getJobId(),""+ task.getIndexCode()};

解决方案 #

问题根源 #

把包含转义引号的长字符串作为参数传递给bash脚本时，bash会解析引号，导致--class后的类名被额外引号包裹，Spark无法识别带引号的类名，从而抛出加载失败错误。

修复步骤 #

1. 修改Java代码：移除配置字符串中所有转义引号，Java通过数组传递参数时，单个元素会被bash当作完整参数，不需要额外引号分隔：

String CONF = "--class com.at.es_parent_child.SegmentIcebergEsV2 --master yarn --deploy-mode client --queue llap --num-executors 3 --driver-memory 1024m --executor-memory 1024m --executor-cores 4 --name [564889711]es_parent_child.[0].1668574353481 --conf spark.executor.extraClassPath=/etc/hbase/conf --conf spark.driver.extraClassPath=/etc/hbase/conf --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.max.executor.failures=100 --conf spark.rdd.compress=true --conf spark.sql.debug.maxToStringFields=2000 --conf spark.sql.hive.convertMetastoreParquet=false --conf spark.default.parallelism=50 --conf spark.debug.maxToStringFields=2000 --conf hbase.defaults.for.version.skip=true --conf spark.yarn.executor.memoryOverhead=1024 --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.type=hive --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.iceberg.type=hive --conf spark.sql.adaptive.coalescePartitions.enabled=true --files /etc/hbase/conf/hbase-site.xml,/usr/hdp/current/hive-client/conf/hive-site.xml";

2. 修改Bash脚本：将硬编码的CONF替换为传入的$1，并用双引号包裹确保参数解析正确，同时给后续参数也加上双引号避免空格拆分：

CONF="$1"
echo "--------------------------------------------"
echo "$CONF"
echo "--------------------------------------------"

sudo -u cdpcore /bin/sh /build/iceberg/spark-3.0.1-bin-hadoop2.7/bin/spark-submit $CONF --jars $(echo $JAR_LIB/*.jar | tr ' ' ',') $JAR_MAIN "$2" "$3" "$4" "$5" &

echo sudo -u cdpcore /bin/sh /build/iceberg/spark-3.0.1-bin-hadoop2.7/bin/spark-submit $CONF --jars $(echo $JAR_LIB/*.jar | tr ' ' ',') $JAR_MAIN "$2" "$3" "$4" "$5" &

额外说明 #

如果参数值包含空格，无需在Java代码中额外加引号，Java数组传递的单个参数会被bash完整识别，不会拆分空格。例如带空格的配置可直接写为--conf key="value with space"。

内容的提问来源于stack exchange，提问作者robocon20x