Livy无法接收大JSON参数的问题排查与解决方案咨询

背景 #

我有一个C# API，会向Livy的/batches路由发送HTTP POST请求，Livy会将参数转发给我的Scala Spark Driver。Livy内部会在Spark master的容器中运行spark-submit命令。

请求体示例 #

{
    "file": "/opt/scala-apps/spark-driver-assembly-1.0.0.jar",
    "proxyUser": "X",
    "className": "Y",
    "args": ["a gigantic json"],
    "name": "myTest",
    "conf": {
        "spark.sql.broadcastTimeout": "1500",
        "spark.driver.extraJavaOptions": "-Dlog4j.configuration=file:/opt/scala-apps/livy/conf/log4j.properties -Dguid=111 -Dtimestamp=20240424"
    }
}

问题描述 #

传入Livy的args中的JSON内容非常大，通过Postman模拟请求时，收到错误：

java.io.IOException: Cannot run program "/opt/spark/bin/spark-submit": error=7, Argument list too long

推测Livy服务器内部代码无法处理这么大的单个参数。

测试情况 #

将大JSON拆分为两个字符串放入args数组后，请求可以正常执行，但这种拆分方案不适用于我的业务场景，因此不想采用。

疑问与需求 #

怀疑这是容器shell或操作系统的参数长度限制导致的，需要知道如何排查该配置，同时希望获得合理的解决方案。

补充信息 #

我的Spark Driver采用1主2从的独立集群模式（Spark 3.5版本），相关配置文件如下：

Spark Master配置文件 #

spark.master spark://myServer:7077
spark.sql.caseSensitive false
spark.executor.heartbeatInterval 90000
spark.network.timeout 400000
spark.executor.heartbeat.maxFailures 10
spark.shuffle.registration.timeout 500000
spark.shuffle.push.finalize.timeout 600s
spark.files.fetchTimeout 600s
spark.rpc.lookupTimeout 600s
spark.scheduler.excludeOnFailure.unschedulableTaskSetTimeout 600s
spark.eventLog.enabled true
spark.eventLog.dir file:/opt/spark/logs/spark-events/
spark.history.fs.logDirectory file:/opt/spark/logs/spark-events/
spark.executor.logsDirectory /opt/spark/logs
spark.sql.adaptive.enabled true
spark.sql.adaptive.skewJoin.enabled true
spark.sql.adaptive.localShuffleReader.enabled true
spark.sql.adaptive.coalescePartitions.enabled true
spark.executor.memory 5g
spark.executor.cores 2
spark.driver.memory 8g
spark.driver.cores 4
spark.dynamicAllocation.enabled true
spark.dynamicAllocation.shuffleTracking.enabled true
spark.dynamicAllocation.executorIdleTimeout 600s
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.port 7078
spark.blockManager.port 7087
spark.driver.blockManager.port 7011
spark.shuffle.service.enabled true
spark.shuffle.service.port 7337
spark.submit.deployMode client
spark.worker.cleanup.enabled true

Livy配置文件 #

livy.spark.master = spark://X:7077
livy.spark.deploy-mode = client

# Configure Livy server http request and response header size.
#livy.server.request-header.size = 300000
#livy.server.response-header.size = 300000

livy.server.session.state-retain.sec = 600s
livy.cache-log.size = 1000000
livy.file.local-dir-whitelist = /opt/scala-apps

Docker Compose配置文件 #

version: "3.4"

services:
    spark_master:
        container_name: spark_master
        image: apache/spark:3.5.0
        stdin_open: true
        tty: true
        user: root
        network_mode: host
        environment:
            - TZ=Asia/Baghdad
            - SPARK_PUBLIC_DNS=X
        restart: unless-stopped
        volumes:
            - volumes
        ports:
            - ports
        entrypoint: y

    spark_worker:
        container_name: spark_worker
        image: apache/spark:3.5.0
        stdin_open: true
        tty: true
        user: root
        network_mode: host
        environment:
            - TZ=Asia/Baghdad
            - SPARK_MASTER_ADDRESS=spark://X:7077
            - SPARK_WORKER_PORT=7087
            - SPARK_PUBLIC_DNS=
        restart: unless-stopped
        volumes:
            - volumes
        ports:
            - ports
        entrypoint: z

内容的提问来源于stack exchange，提问作者Giovanna Bueno