Google Dataflow资源文件查找失败，存在双重dataflow目录异常

I've seen other users report this exact double dataflow directory mismatch in Dataflow worker instances around mid-2018. Here's how to resolve it, plus context on what's going wrong:

Root Cause #

This is a temporary deployment path bug in Dataflow workers from that timeframe. Your JAR (and embedded resources) gets placed one level deeper than the runtime expects—landing in /var/opt/google/dataflow/dataflow/ instead of the anticipated /var/opt/google/dataflow/. The runtime tries to look in the shallower path, hence the FileNotFoundException.

Practical Solutions #

1. Use getResourceAsStream() (Recommended) #

The Class.getResource().getFile() method is unreliable for JAR-internal resources in distributed environments like Dataflow. Instead, load the resource directly as an input stream to bypass file system path issues entirely:

try (InputStream resourceStream = WordCount.class.getResourceAsStream("/data.json");
     Scanner scanner = new Scanner(resourceStream).useDelimiter("\\Z")) {
    String jsonStr = scanner.next();
    System.out.println("====================================================");
    System.out.println(jsonStr);
    System.out.println("====================================================");
} catch (Exception e) {
    e.printStackTrace();
}

This method reads the resource straight from the JAR, so it doesn't care about the worker's directory structure.

2. Upgrade Your Beam/Dataflow SDK #

This path mismatch bug was fixed in later versions of the SDK. Upgrading to a release post-2018 will permanently resolve the issue without code changes.

3. Verify Build Configuration (Quick Check) #

Double-check your build tool ensures the resources directory is properly packaged:

• For Maven: Confirm src/main/resources is recognized as a resource directory (this is enabled by default, but it's worth verifying your pom.xml hasn't been modified)
• For Gradle: Ensure sourceSets.main.resources.srcDirs = ['src/main/resources'] is set in your build script

4. Brittle Workaround (Not Recommended for Production) #

If you need a short-term fix while planning an upgrade, you can manually adjust the path to account for the nested directory. Note this is fragile (worker paths might change):

// Extract the JAR name from the current code source path
String jarPath = WordCount.class.getProtectionDomain().getCodeSource().getLocation().getPath();
String jarName = new File(jarPath).getName();
// Build the corrected path with the extra "dataflow" directory
File correctedJarLocation = new File("/var/opt/google/dataflow/dataflow/" + jarName);
// You'd then need to extract the resource from this JAR manually

Final Note #

Reporting this on GitHub is a great move—platform-level path bugs like this benefit from community visibility to ensure they get prioritized and fixed.

内容的提问来源于stack exchange，提问作者Jun Ying