从Savepoint恢复Flink作业时遭遇NoSuchMethodError问题求助

报错堆栈 #

java.lang.NoSuchMethodError: org.apache.commons.cli.CommandLine.hasOption(Lorg/apache/commons/cli/Option;)Z
    at org.apache.flink.client.cli.CliFrontendParser.createSavepointRestoreSettings(CliFrontendParser.java:631) ~[flink-dist-1.16.1.jar:1.16.1]
    at org.apache.flink.client.cli.ProgramOptions.<init>(ProgramOptions.java:119) ~[flink-dist-1.16.1.jar:1.16.1]
    at org.apache.flink.client.cli.ProgramOptions.create(ProgramOptions.java:192) ~[flink-dist-1.16.1.jar:1.16.1]
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:230) ~[flink-dist-1.16.1.jar:1.16.1]
    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1087) ~[flink-dist-1.16.1.jar:1.16.1]
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1165) ~[flink-dist-1.16.1.jar:1.16.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_372]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_372]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
    at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist-1.16.1.jar:1.16.1]
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1165) [flink-dist-1.16.1.jar:1.16.1]

问题背景 #

使用Apache Flink 1.16.1版本，通过fromSavepoint选项部署作业时触发上述错误。初步判断是commons-cli版本不匹配导致——作业部署需要commons-cli至少1.5.0版本，但当前环境无法满足。不过检查Maven依赖后，并未发现低于1.5.0版本的commons-cli。

相关Flink源码 #

报错发生在CliFrontendParser.createSavepointRestoreSettings方法的第631行：

public static SavepointRestoreSettings createSavepointRestoreSettings(CommandLine commandLine) {
    if (commandLine.hasOption(SAVEPOINT_PATH_OPTION.getOpt())) {
        String savepointPath = commandLine.getOptionValue(SAVEPOINT_PATH_OPTION.getOpt());
        boolean allowNonRestoredState =
                commandLine.hasOption(SAVEPOINT_ALLOW_NON_RESTORED_OPTION.getOpt());
        final RestoreMode restoreMode;
        if (commandLine.hasOption(SAVEPOINT_RESTORE_MODE)) {
            restoreMode =
                    ConfigurationUtils.convertValue(
                            commandLine.getOptionValue(SAVEPOINT_RESTORE_MODE),
                            RestoreMode.class);
        } else {
            restoreMode = SavepointConfigOptions.RESTORE_MODE.defaultValue();
        }
        return SavepointRestoreSettings.forPath(
                savepointPath, allowNonRestoredState, restoreMode);
    } else {
        return SavepointRestoreSettings.none();
    }
}

注意到第626行的commandLine.hasOption(SAVEPOINT_PATH_OPTION.getOpt())可以正常执行，但第631行的commandLine.hasOption(SAVEPOINT_RESTORE_MODE)却抛出异常，二者的区别在于前者调用了.getOpt()方法获取选项字符串，后者直接传入Option对象。

验证代码及疑问 #

为验证环境和代码逻辑，运行了以下Scala代码，并未抛出异常：

val opt = new Option("rm", "restoreMode", true, "Defines how should we restore from the given savepoint. Supported options: " + "[claim - claim ownership of the savepoint and delete once it is" + " subsumed, no_claim (default) - do not claim ownership, the first" + " checkpoint will not reuse any files from the restored one, legacy " + "- the old behaviour, do not assume ownership of the savepoint files," + " but can reuse some shared files.")

val cl = new CommandLine.Builder().build()

if (cl.hasOption(opt.getOpt)) {
 logger.error("l!")
} else {
  logger.error("p!")
}

if (cl.hasOption(opt)) {
  logger.error("y!")
}
else {
  logger.error("x!")
}

既然这段代码能正常运行，为何Flink在相同场景下会报错？该如何解决这个问题？

解决方案 #

• 检查集群类路径中的依赖冲突：Maven依赖中没有低版本commons-cli不代表集群运行环境里没有。比如报错中出现的flink-shaded-hadoop-2-uber-2.8.3-10.0.jar可能内置了低版本commons-cli，导致类加载时优先加载旧版本。CommandLine.hasOption(Option)方法是commons-cli 1.4及以后才新增的，旧版本没有这个方法，因此触发错误。
• 强制指定commons-cli版本并排除冲突依赖：在Maven中明确指定commons-cli 1.5.0版本，并排除其他依赖（如flink-dist、flink-shaded-hadoop-2-uber）中自带的commons-cli，避免版本冲突：

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-cli</artifactId>
    <version>1.5.0</version>
    <scope>compile</scope>
</dependency>

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-dist</artifactId>
    <version>1.16.1</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-cli</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-shaded-hadoop-2-uber</artifactId>
    <version>2.8.3-10.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-cli</artifactId>
        </exclusion>
    </exclusions>
</dependency>

• 调整作业提交的类路径顺序：提交作业时，确保高版本commons-cli的jar包排在类路径最前面，让类加载器优先加载正确版本的类。
• 使用shade插件重命名commons-cli包：通过Maven的maven-shade-plugin打包作业时，将commons-cli的类重命名到自定义包下，彻底避免与其他依赖的类冲突：

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <relocations>
                    <relocation>
                        <pattern>org.apache.commons.cli</pattern>
                        <shadedPattern>your.custom.package.shaded.commons.cli</shadedPattern>
                    </relocation>
                </relocations>
            </configuration>
        </execution>
    </executions>
</plugin>

内容的提问来源于stack exchange，提问作者lifeisshort