AWS Glue Spark作业调用Job.init报错，寻求调试解决方法

错误详情 #

Traceback (most recent call last):
  File "C:\opt\inbound_db.py", line 31, in <module>
    glue_spark.job.init(
  File "C:\opt\Anaconda3\Lib\site-packages\awsglue\job.py", line 42, in init
    self._job.init(job_name, self._glue_context._glue_scala_context, args)
  File "C:\opt\Anaconda3\Lib\site-packages\py4j\java_gateway.py", line 1322, in __call__
    return_value = get_return_value(
                   ^^^^^^^^^^^^^^^^^
  File "C:\opt\Anaconda3\Lib\site-packages\pyspark\errors\exceptions\captured.py", line 169, in deco
    return f(*a, **kw)
           ^^^^^^^^^^^
  File "C:\opt\Anaconda3\Lib\site-packages\py4j\protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.amazonaws.services.glue.util.Job.init.
: java.lang.NoClassDefFoundError: com/typesafe/config/ConfigMergeable
    at com.amazonaws.services.glue.util.Job$.init(Job.scala:96)
    at com.amazonaws.services.glue.util.Job.init(Job.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:75)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.ClassNotFoundException: com.typesafe.config.ConfigMergeable
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    ... 14 more

23/11/14 22:12:22 INFO SparkContext: Invoking stop() from shutdown hook
23/11/14 22:12:22 INFO SparkContext: SparkContext is stopping with exitCode 0.

Process finished with exit code 1

核心问题分析 #

堆栈明确指向java.lang.NoClassDefFoundError: com/typesafe/config/ConfigMergeable，本质是JVM找不到typesafe配置库的核心类，属于依赖缺失或版本不兼容问题，常见于本地开发环境配置错误，或Glue集群作业缺少额外依赖。

调试步骤 #

• 检查本地AWS Glue依赖版本
  执行pip show awsglue查看当前安装的包版本，不同Glue版本（如2.0/3.0/4.0）对应不同的typesafe config依赖版本，确保两者匹配。

• 补充缺失的依赖包

  1. 下载对应版本的typesafe-config.jar（推荐适配Glue 3.0的1.4.2版本）
  2. 本地开发时：
     • 将jar包放入Anaconda环境的spark/jars目录，或启动脚本中添加--jars /path/to/typesafe-config-1.4.2.jar参数
  3. Glue集群运行时：
     • 将jar包上传至S3存储桶，然后在Glue作业的作业参数中添加--extra-jars s3://你的存储桶路径/typesafe-config-1.4.2.jar

• 验证本地Spark环境配置

  • 确保本地Spark版本与Glue集群版本一致（如Glue 3.0对应Spark 3.1）
  • 检查Spark配置中的spark.driver.extraClassPath，确认已包含typesafe config jar的路径

• 排查虚拟环境依赖冲突

  • 若使用Anaconda虚拟环境，删除现有环境重新创建，仅安装awsglue及必要依赖：pip install awsglue，避免其他第三方包（如手动安装的pyspark）与awsglue自带依赖冲突

• 集群作业权限验证

  • 若本地调试正常但集群报错，确认Glue作业的IAM角色拥有读取S3依赖包路径的权限
  • 检查S3中jar包的存储路径是否正确，无拼写错误

内容的提问来源于stack exchange，提问作者daturm girl