Dataflow SDK 1.x迁移至2.x后Eclipse运行正常，Maven打包运行报错求助

I’ve run into this exact issue when migrating from Dataflow 1.x to Beam 2.x—your code works smoothly in Eclipse but breaks when running a fat jar built via the assembly plugin. Let’s break down why this happens and how to fix it:

Root Cause #

The warning about PipelineOptions.filesToStage is a critical clue. When you use assembly:single to bundle everything into one fat jar, Beam’s default staging logic only detects this single jar file instead of the individual dependencies it expects. This leads to missing classes or resources at runtime, which triggers the exception (even though your log snippet cuts off, this is the most common culprit here).

Solutions #

1. Use the Official Beam Maven Plugin (Recommended) #

Beam provides a dedicated maven plugin that handles packaging for its runners perfectly, avoiding the filesToStage issue entirely. Here’s how to set it up:

• Update your pom.xml to include the Beam plugin:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-maven-plugin</artifactId>
      <version>${beam.version}</version>
      <executions>
        <execution>
          <goals>
            <goal>package</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

• Build your jar with:

mvn clean package

• Run the jar using:

java -jar your-project-name-1.0-SNAPSHOT.jar [your pipeline options]

This plugin correctly stages all necessary dependencies and configures the classpath properly for Beam runners.

2. Explicitly Configure filesToStage (If You Must Use Assembly Plugin) #

If you need to stick with the assembly plugin, you can manually set the filesToStage option to include all elements from your classpath. Add this code when creating your PipelineOptions:

import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;

public class YourPipeline {
  public static void main(String[] args) {
    PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
    
    // Explicitly set filesToStage to all classpath entries
    List<String> filesToStage = new ArrayList<>();
    for (URL url : ((URLClassLoader) Thread.currentThread().getContextClassLoader()).getURLs()) {
      filesToStage.add(url.getPath());
    }
    options.setFilesToStage(filesToStage);
    
    // Rest of your pipeline code...
  }
}

This forces Beam to stage every file in your classpath, including the fat jar and any other dependencies.

3. Verify Dependency Scopes in pom.xml #

Double-check that all Beam-related dependencies are set to compile scope (not provided). In Eclipse, some dependencies might be included via the IDE, but the assembly plugin won’t include provided dependencies in the fat jar. For example:

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-core</artifactId>
  <version>${beam.version}</version>
  <scope>compile</scope> <!-- Ensure this is set to compile -->
</dependency>

Try these steps—most likely switching to the Beam Maven plugin will resolve the issue immediately. If not, checking your dependency scopes or explicitly setting filesToStage should do the trick.

内容的提问来源于stack exchange，提问作者Abhishek Verma