You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

PySpark训练MultilayerPerceptronClassifier时ArrayIndexOutOfBoundsException排查

avatar
安全阿岳
2026-7-27
问题描述

我正在开展一个PySpark项目，尝试使用fit方法将MultilayerPerceptronClassifier模型拟合到文本数据中，采用MLlib提供的Word2Vec模型提取特征。但调用fit方法时，持续触发ArrayIndexOutOfBoundsException错误，具体错误信息如下：

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-1-d46ecefd3281> in <module>
     41 # Use Word2Vec to generate word embeddings
     42 word2Vec_pipeline = Pipeline(stages=[tokenizer, word2Vec, labelIndexer, mlp])
---&gt; 43 word2Vec_model = mlp.fit(trainingData_word2Vec)
     44 word2Vec_predictions = word2Vec_model.transform(testData_word2Vec)
     45 

~\Documents\spark-3.3.1-bin-hadoop3\python\pyspark\ml\base.py in fit(self, dataset, params)
    203                 return self.copy(params)._fit(dataset)
    204             else:
--&gt; 205                 return self._fit(dataset)
    206         else:
    207             raise TypeError(

~\Documents\spark-3.3.1-bin-hadoop3\python\pyspark\ml\wrapper.py in _fit(self, dataset)
    381 
    382     def _fit(self, dataset: DataFrame) -&gt; JM:
--&gt; 383         java_model = self._fit_java(dataset)
    384         model = self._create_model(java_model)
    385         return self._copyValues(model)

~\Documents\spark-3.3.1-bin-hadoop3\python\pyspark\ml\wrapper.py in _fit_java(self, dataset)
    378 
    379         self._transfer_params_to_java()
--&gt; 380         return self._java_obj.fit(dataset._jdf)
    381 
    382     def _fit(self, dataset: DataFrame) -&gt; JM:

~\anaconda3\lib\site-packages\py4j\java_gateway.py in __call__(self, *args)
   1319 
   1320         answer = self.gateway_client.send_command(command)
-&gt; 1321         return_value = get_return_value(
   1322             answer, self.gateway_client, self.target_id, self.name)
   1323 

~\Documents\spark-3.3.1-bin-hadoop3\python\pyspark\sql\utils.py in deco(*a, **kw)
    188     def deco(*a: Any, **kw: Any) -&gt; Any:
    189         try:
--&gt; 190             return f(*a, **kw)
    191         except Py4JJavaError as e:
    192             converted = convert_exception(e.java_exception)

~\anaconda3\lib\site-packages\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--&gt; 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling o181.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 1 times, most recent failure: Lost task 0.0 in stage 15.0 (TID 39) (DESKTOP-HILGIEG executor driver): java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.apache.spark.ml.ann.DataStacker.$anonfun$stack$4(Layer.scala:665)
    at org.apache.spark.ml.ann.DataStacker.$anonfun$stack$4$adapted(Layer.scala:664)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at org.apache.spark.ml.ann.DataStacker.$anonfun$stack$3(Layer.scala:664)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:224)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:302)
    at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1518)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1445)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1509)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1332)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:327)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:136)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2228)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2249)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2268)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2293)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1274)
    at org.apache.spark.mllib.optimization.LBFGS$.runLBFGS(LBFGS.scala:195)
    at org.apache.spark.mllib.optimization.LBFGS.optimizeWithLossReturned(LBFGS.scala:154)
    at org.apache.spark.ml.ann.FeedForwardTrainer.train(Layer.scala:855)
    at org.apache.spark.ml.classification.MultilayerPerceptronClassifier.$anonfun$train$1(MultilayerPerceptronClassifier.scala:228)
    at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
    at scala.util.Try$.apply(Try.scala:213)
    at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
    at org.apache.spark.ml.classification.MultilayerPerceptronClassifier.train(MultilayerPerceptronClassifier.scala:184)
    at org.apache.spark.ml.classification.MultilayerPerceptronClassifier.train(MultilayerPerceptronClassifier.scala:93)
    at org.apache.spark.ml.Predictor.fit(Predictor.scala:151)
    at org.apache.spark.ml.Predictor.fit(Predictor.scala:115)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.apache.spark.ml.ann.DataStacker.$anonfun$stack$4(Layer.scala:665)
    at org.apache.spark.ml.ann.DataStacker.$anonfun$stack$4$adapted(Layer.scala:664)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at org.apache.spark.ml.ann.DataStacker.$anonfun$stack$3(Layer.scala:664)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:224)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:302)
    at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1518)
    at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1445)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1509)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1332)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:327)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:136)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more

问题原因分析

这个错误的核心是MLP输入层神经元数量与Word2Vec输出的特征维度不匹配：

  • MultilayerPerceptronClassifier的layers参数第一个值必须严格等于输入特征的维度，否则会在数据堆叠时触发数组越界
  • Word2Vec默认向量维度为100，如果你的MLP输入层设置的不是100，就会导致该异常
  • 即使数据没有空值，若存在空分词结果（比如空白文本），Word2Vec会输出全0向量，但MLP层配置错误仍会触发异常

解决方法

  • 匹配特征维度与MLP输入层：
    1. 确认Word2Vec的向量维度：如果显式设置了vectorSize参数，使用该值；未设置则默认是100
    2. 调整MLP的layers参数，确保第一个元素等于Word2Vec的向量维度。例如Word2Vec维度为100，分类任务有2个类别，layers应设置为[100, 64, 2]
  • 处理空分词结果：
    过滤掉分词后为空的文本行，示例代码：
    from pyspark.sql.functions import size, col

trainingData_word2Vec = trainingData_word2Vec.filter(size(col("tokens")) > 0)
  • 检查Pipeline配置（若使用Pipeline）：
    确保MultilayerPerceptronClassifier的featuresCol指向Word2Vec的输出列（默认列名为word2vec），避免直接使用原始分词列作为输入

内容的提问来源于stack exchange，提问作者Ibtissam Youb

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.07.27 17:32:02