升级Drill至1.14.0+时YARN作业出现JniBasedUnixGroupsMapping类缺失异常

环境与配置 #

• 部署架构：Drill集群部署在Hadoop EMR核心节点
• 认证方式：PAM认证，配置如下：

security.user.auth: {
            enabled: true,
            packages += "org.apache.drill.exec.rpc.user.security",
            impl: "pam4j",
            pam_profiles: [ "sudo", "login" ]
     }

• 作业提交方式：通过YARN提交，YARN应用通过JDBC连接Drillbit生成结果
• EMR版本：5.36.1

问题现象 #

• Drill 1.6.0版本时，Drill+YARN作业运行正常
• 升级至1.14.0及以上版本后，仅YARN作业失败，常规JDBC sqlline连接无异常
• 禁用security.user.auth配置后，YARN作业恢复正常

异常堆栈信息 #

Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at oadd.org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
at oadd.org.apache.hadoop.security.Groups.(Groups.java:79)
at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
at oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
at oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
at oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
at oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
at oadd.org.apache.drill.exec.rpc.user.UserClient.prepareSaslHandshake(UserClient.java:452)
at oadd.org.apache.drill.exec.rpc.ConnectionMultiListener$HandshakeSendHandler.success(ConnectionMultiListener.java:160)
... 33 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at oadd.org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
... 41 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class oadd.org.apache.hadoop.security.JniBasedUnixGroupsMapping
at oadd.org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.(JniBasedUnixGroupsMappingWithFallback.java:39)
... 46 more

内容的提问来源于stack exchange，提问作者Anish