Payara服务器中使用DL4J时matmul操作执行失败问题排查

核心问题根源 #

这个错误的直接触发点是NativeOpExecutioner.getCustomOperations方法中的数组越界，结合独立应用正常、Payara服务器上异常的现象，主要原因集中在以下几点：

1. 类加载器冲突（最可能） #

你将DL4J/ND4J依赖设为provided，意味着依赖由Payara服务器提供而非应用包自带。但：

• Payara自身可能包含冲突的ND4J/DL4J类，或服务器类加载器优先级导致应用加载了不兼容的类版本
• ND4J的Native库加载依赖上下文类加载器，应用服务器的类加载机制会干扰Native库的正确初始化，导致自定义操作（如matmul）的元数据加载不全，触发数组越界

2. Native库加载异常 #

虽然初始化日志显示加载了CpuBackend，但在应用服务器环境下：

• Native库的提取路径可能受Payara安全限制或目录权限影响，导致部分操作实现未正确加载
• OpenMP线程配置与Payara线程池冲突，影响Native操作的执行上下文

3. 老旧版本的已知bug #

你使用的DL4J/ND4J 1.0.0-M2是2021年的版本，该版本存在应用服务器类加载相关的已知问题，后续版本已修复此类Native操作执行时的数组越界bug。

验证与修复步骤 #

• 移除provided依赖：将DL4J/ND4J的依赖scope改为compile，让应用自带完整依赖包，避免与服务器类冲突
• 升级版本：将DL4J/ND4J版本升级到最新稳定版（如1.0.0-M3.1或更高），修复旧版本的类加载和Native操作bug
• 调整类加载器配置：在Payara中为应用配置delegate="false"，强制应用优先加载自身的类而非服务器提供的类
• 检查Native库权限：确保Payara运行用户对ND4J提取Native库的临时目录（通常为java.io.tmpdir）有读写权限

问题原始信息 #

问题描述 #

在Payara服务器上部署的应用中，MultiLayerNetwork模型加载正常，但对INDArray执行推理请求时抛出RuntimeException。独立测试应用中模型加载与推理均正常，服务器环境下出现异常。

初始化日志 #

org.nd4j.linalg.factory.Nd4jBackend - Loaded [CpuBackend] backend
org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 4
org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - Binary level Generic x86 optimization level AVX512
org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 8
org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [16]; Memory: [7,1GB]
org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [OPENBLAS]

org.nd4j.linalg.cpu.nativecpu.CpuBackend - Backend build information:
 GCC: "7.5.0"
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS

org.deeplearning4j.nn.multilayer.MultiLayerNetwork - Starting MultiLayerNetwork with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]

推理错误日志 #

ERROR org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner - Failed to execute op matmul. Attempted to execute with 2 inputs, 1 outputs, 2 targs,0 bargs and 3 iargs. Inputs: [(FLOAT,[1,9],c), (FLOAT,[9,400],f)]. Outputs: [(FLOAT,[1,400],f)]. tArgs: [1.0, 0.0]. iArgs: [0, 0, 0]. bArgs: -. Op own name: "81f000d9-6011-4d1f-adc4-af57fb7d11e6" - Please see above message (printed out from c++) for a possible cause of error.

堆栈跟踪 #

java.lang.RuntimeException: Op [matmul] execution failed
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1561) ~[nd4j-native-1.0.0-M2.jar:?]
        at org.nd4j.linalg.factory.Nd4j.exec(Nd4j.java:6522) ~[nd4j-api-1.0.0-M2.jar:?]
        at org.nd4j.linalg.api.blas.impl.BaseLevel3.gemm(BaseLevel3.java:62) ~[nd4j-api-1.0.0-M2.jar:?]
        at org.nd4j.linalg.api.ndarray.BaseNDArray.mmuli(BaseNDArray.java:3194) ~[nd4j-api-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.layers.BaseLayer.preOutputWithPreNorm(BaseLayer.java:322) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.layers.BaseLayer.preOutput(BaseLayer.java:295) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.layers.BaseLayer.activate(BaseLayer.java:343) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.layers.AbstractLayer.activate(AbstractLayer.java:262) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.outputOfLayerDetached(MultiLayerNetwork.java:1341) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.output(MultiLayerNetwork.java:2453) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.output(MultiLayerNetwork.java:2416) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.output(MultiLayerNetwork.java:2407) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.output(MultiLayerNetwork.java:2394) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.output(MultiLayerNetwork.java:2490) ~[deeplearning4j-nn-1.0.0-M2.jar:?]
        at com.[****].lambda$nnRegression$1(myCustomJavaClass.java:264)

Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getCustomOperations(NativeOpExecutioner.java:1365) ~[nd4j-native-1.0.0-M2.jar:?]
        at org.nd4j.linalg.api.ops.DynamicCustomOp.opHash(DynamicCustomOp.java:392) ~[nd4j-api-1.0.0-M2.jar:?]
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1900) ~[nd4j-native-1.0.0-M2.jar:?]
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1540) ~[nd4j-native-1.0.0-M2.jar:?]
        ... 139 more

测试代码 #

MultiLayerNetwork net = MultiLayerNetwork.load(nn, false);
List<Integer> obsList = new ArrayList<>();

obsList.add(1);obsList.add(18);
obsList.add(1);obsList.add(24);
obsList.add(1);obsList.add(15);
obsList.add(1);obsList.add(13);
obsList.add(2);

int[] obsArray = obsList.stream().mapToInt(Integer::intValue).toArray();
int[][] flat = new int[][] { obsArray };

INDArray test = Nd4j.create(flat);
INDArray y = net.output(test); // <---- 错误发生在此处

依赖配置（POM片段） #

<dependency>
    <groupId>org.deeplearning4j</groupId>
    <artifactId>deeplearning4j-core</artifactId>
    <version>1.0.0-M2</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.nd4j</groupId>
    <artifactId>nd4j-native-platform</artifactId>
    <version>1.0.0-M2</version>
    <scope>provided</scope>
</dependency>

运行环境 #

• 操作系统：CentOS 8
• 应用服务器：Payara Server 5.193.1 #badassfish（build 275）

内容的提问来源于stack exchange，提问作者ThomasVoss