如何结合Terraform与Docker为EMR Serverless安装内部JAR包

方案1：Terraform + S3存储依赖包，任务运行时加载 #

这是最直接的适配方案，无需混合Docker和Terraform：

• 将内部Java/Scala JAR包上传至AWS S3桶，确保EMR Serverless执行角色拥有该桶的读取权限
• 在Terraform配置EMR Serverless应用时，通过spark.submit.parameters或hadoop.submit.parameters指定依赖路径：

  resource "aws_emrserverless_application" "spark_app" {
    name           = "my-spark-app"
    release_label  = "emr-6.10.0"
    type           = "SPARK"
  
    initial_capacity {
      key = "DRIVER"
      value {
        worker_count = 1
        worker_configuration {
          cpu    = "2vCPU"
          memory = "4GB"
        }
      }
    }
  
    maximum_capacity {
      cpu    = "16vCPU"
      memory = "32GB"
    }
  
    # 配置Spark任务参数，指定S3中的JAR依赖
    configuration_overrides {
      application_configuration {
        classification = "spark-defaults"
        properties = {
          "spark.jars" = "s3://your-bucket/path/to/your-internal.jar,s3://your-bucket/path/to/other-dependency.jar"
        }
      }
    }
  }
  

• 提交任务时，也可在aws_emrserverless_job_run资源中指定spark.jars参数，覆盖应用级配置

方案2：自定义EMR Serverless镜像 + Terraform配置参数 #

若必须使用Docker镜像，可通过以下方式同时实现参数配置：

• 构建自定义Docker镜像时，将内部JAR包复制到镜像指定目录（如/opt/spark/jars），确保镜像遵循EMR Serverless规范（基于官方EMR Serverless镜像修改）
• 在Terraform中配置EMR Serverless应用时，指定自定义镜像地址，同时正常配置初始容量、最大内存等参数：

  resource "aws_emrserverless_application" "custom_spark_app" {
    name           = "my-custom-spark-app"
    release_label  = "emr-6.10.0"
    type           = "SPARK"
  
    image_configuration {
      image_uri = "your-ecr-repo/custom-emr-serverless-image:latest"
    }
  
    initial_capacity {
      key = "DRIVER"
      value {
        worker_count = 1
        worker_configuration {
          cpu    = "2vCPU"
          memory = "4GB"
        }
      }
    }
  
    maximum_capacity {
      cpu    = "16vCPU"
      memory = "32GB"
    }
  }
  

• 注意：自定义镜像必须基于官方提供的public.ecr.aws/emr-serverless/spark/emr-6.10.0:latest类镜像构建，避免兼容性问题

方案3：Terraform配置 + 启动脚本安装依赖 #

若依赖包无法提前上传到S3，可通过任务启动脚本动态拉取：

• 在Terraform的任务配置中，添加spark.submit.driverExtraClassPath或spark.executor.extraClassPath，同时指定启动脚本下载并加载JAR：

  resource "aws_emrserverless_job_run" "spark_job" {
    application_id = aws_emrserverless_application.spark_app.id
    execution_role_arn = aws_iam_role.emr_serverless_exec.arn
  
    job_driver {
      spark_submit {
        entry_point = "s3://your-bucket/path/to/main.jar"
        entry_point_arguments = ["arg1", "arg2"]
        spark_submit_parameters = "--conf spark.jars=/tmp/internal.jar --conf spark.driver.extraClassPath=/tmp/internal.jar --conf spark.executor.extraClassPath=/tmp/internal.jar"
      }
    }
  
    configuration_overrides {
      application_configuration {
        classification = "spark-defaults"
        properties = {
          "spark.executorEnv.PYSPARK_PYTHON" = "/usr/bin/python3"
        }
      }
      monitoring_configuration {
        s3_monitoring_configuration {
          log_uri = "s3://your-bucket/logs/"
        }
      }
    }
  
    # 添加启动脚本，从内部存储拉取JAR到临时目录
    setup_script_configuration {
      script_uri = "s3://your-bucket/path/to/download-jar.sh"
      executable = "/bin/bash"
    }
  }
  

• download-jar.sh示例内容：

  #!/bin/bash
  # 从内部文件服务器或其他存储拉取JAR到/tmp目录
  curl -o /tmp/internal.jar http://your-internal-server/path/to/internal.jar
  

内容的提问来源于stack exchange，提问作者user20505247