Jar包已包含目标类仍触发ClassNotFoundException的Spark应用排查求助

Let's break down this confusing issue step by step—your key clues here are the conflicting results between jar tvf (seeing the class) and jar xf (missing the directory), plus the fact other unzip tools can see the content. Here's what's likely going on and how to fix it:

1. JDK jar Command Compatibility Issues #

The most immediate culprit here is probably the version of the jar tool you're using. Older JDK versions (especially pre-JDK 8u200 or JDK 7) have known issues handling multi-release JARs (MRJARs) or JARs with nested directory structures that newer tools handle correctly. PostgreSQL's driver JAR (starting from version 42.x) uses MRJAR format to support multiple JDK versions, which can trip up older jar commands during extraction.

How to Confirm: #

• Check your JDK version: Run java -version and jar -version to see if you're on an older release.
• Use unzip instead of jar xf to extract your shaded JAR:

  unzip myShadedJar.jar -d temp_extract
  ls temp_extract/org/postgresql/ds/
  

  You should see PGSimpleDataSource.class here if the JAR is intact.
• Try using a newer JDK's jar tool (JDK 11+) to extract the file—it should correctly pull the org/postgresql directory.

2. Spark YARN Classloader Isolation #

Even if the class exists in your JAR, Spark's default classloading strategy on YARN might be preventing Hikari from finding it. Spark prioritizes its own bundled dependencies over user JARs by default, and if Hikari is using the system classloader (instead of Spark's application classloader), it won't see your shaded PostgreSQL classes.

How to Confirm & Fix: #

• Add these flags to your spark-submit command to force Spark to prioritize your application's classes:

  --conf spark.driver.userClassPathFirst=true \
  --conf spark.executor.userClassPathFirst=true
  

• Debug the classloader in your code: Right before creating the HikariDataSource, add these lines to check which classloader is being used:

  println("PGSimpleDataSource classloader: " + classOf[org.postgresql.ds.PGSimpleDataSource].getClassLoader())
  println("Current thread context classloader: " + Thread.currentThread().getContextClassLoader())
  

  If they're different, that's the isolation issue at play.

3. Edge Case: Maven Shade Plugin Hidden Conflicts #

While your shade config doesn't exclude PostgreSQL, it's possible the plugin hit a silent conflict when merging JARs (though the log says it included the dependency). For example, if another dependency has overlapping files in the org/postgresql path, the plugin might have overwritten or skipped some classes.

How to Confirm: #

• Open your shaded JAR directly in an IDE like IntelliJ IDEA—navigate to the org/postgresql/ds directory and verify PGSimpleDataSource.class exists.
• Re-run the Maven package command with debug logs enabled to check for warnings about PostgreSQL:

  mvn clean package -X | grep -i postgres
  

  Look for any lines about skipped classes or merge conflicts.

Final Note on Missing Directory During Extraction #

The fact that other unzip tools can see the org/postgresql directory but jar xf cannot is not normal—it's purely a tool compatibility issue, not a problem with your shaded JAR. The JAR itself is intact (as proven by jar tvf and other tools), so this extraction quirk isn't the root cause of the runtime ClassNotFoundException.

内容的提问来源于stack exchange，提问作者b15