如何通过AWS CLI更新AWS Glue作业脚本并保留数据源属性？

当前实现方法 #

• 为每次修改的Python脚本添加版本号，上传至S3：

  aws s3 cp aws_glue_script_v1.0.3_1.py s3://mytestcicdglue/glue-scripts/aws_glue_script_v1.0.3_1.py
  

• 下载Glue作业的JSON配置骨架文件，修改其中的脚本S3路径后更新作业：

  aws glue update-job --job-name glue-cicd-test --job-update file://glue_job_updated_sample.json
  

当前方法的问题 #

采用此方式时，上传新脚本文件后，Glue作业会自动将现有旧代码脚本替换为新版本，无法保留原有的数据源属性配置。

疑问 #

是否存在其他解决方案，能够在更新Glue作业脚本的同时，保留数据源的所有属性？

示例配置文件（glue_job_updated_sample.json） #

{
  "JobMode": "VISUAL",
  "JobRunQueuingEnabled": false,
  "Description": "",
  "Role": "arn:aws:iam::xxxxx:role/Glue-cicd-test-role",
  "ExecutionProperty": {
    "MaxConcurrentRuns": 1
  },
  "Command": {
    "Name": "gluestreaming",  // 此处需修改文件名
    "ScriptLocation": "s3://mytestcicdglue/glue-scripts/aws_glue_script_v1.0.3_1.py",
    "PythonVersion": "3"
  },
  "DefaultArguments": {
    "--enable-metrics": "true",
    "--enable-spark-ui": "true",
    "--extra-py-files": "s3://aws-glue-studio-transforms-xxxxx-prod-ap-southeast-1/misc/openpyxl.zip,s3://aws-glue-studio-transforms-xxxxxx-prod-ap-southeast-1/misc/et_xmlfile.zip",
    "--spark-event-logs-path": "s3://aws-glue-assets-xxxxx-ap-southeast-1/sparkHistoryLogs/",
    "--enable-job-insights": "true",
    "--enable-observability-metrics": "false",
    "--enable-glue-datacatalog": "true",
    "--job-bookmark-option": "job-bookmark-disable",
    "--job-language": "python",
    "--TempDir": "s3://aws-glue-assets-xxxxx-ap-southeast-1/temporary/"
  },
  "MaxRetries": 0,
  "Timeout": 480,
  "WorkerType": "G.1X",
  "NumberOfWorkers": 10,
  "GlueVersion": "5.0",
  "CodeGenConfigurationNodes": {
    "node-1759912958741": {
      "CatalogKafkaSource": {
        "Name": "Apache Kafka",
        "WindowSize": 100,
        "DetectSchema": true,
        "Table": "glue-cicd-test",
        "Database": "testdb",
        "StreamingOptions": {
          "StartingOffsets": "earliest"
        }
      }
    },
    "node-1759909124967": {
      "S3ExcelSource": {
        "Name": "Amazon S3",
        "Paths": [
          "s3://xxxxx/access-list_v1.1.xlsx"
        ],
        "Exclusions": [],
        "Recurse": true,
        "AdditionalOptions": {
          "EnableSamplePath": false,
          "SamplePath": "s3://xxxx/access-list_v1.1.xlsx"
        },
        "OutputSchemas": [
          {
            "Columns": []
          }
        ]
      }
    }
  },
  "ExecutionClass": "STANDARD"
}

内容的提问来源于stack exchange，提问作者aparnagottumukkala