You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

PySpark运行WordCount代码遇序列化IndexError请求排查

avatar
开发者阿泽
2026-8-10
问题：Spark WordCount任务执行时序列化报错

我跟着教程编写了一段Spark WordCount代码，使用spark-submit运行时出现序列化相关的IndexError错误，代码和报错信息如下：

代码实现 

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local").setAppName("WordCount")
sc = SparkContext(conf=conf)

input = sc.textFile("file:///sparkcourse/book.txt")
words = input.flatMap(lambda x: x.split())
wordCounts = words.countByValue()

for word, count in wordCounts.items():
    cleanWord = word.encode('ascii', 'ignore')
    if (cleanWord):
        print(cleanWord.decode() + " " + str(count))

报错信息 

22/11/25 10:31:22 INFO SparkContext: Running Spark version 3.3.1
22/11/25 10:31:22 INFO ResourceUtils: ==============================================================
22/11/25 10:31:22 INFO ResourceUtils: No custom resources configured for spark.driver.
22/11/25 10:31:22 INFO ResourceUtils: ==============================================================
22/11/25 10:31:22 INFO SparkContext: Submitted application: RatingsHistogram
22/11/25 10:31:22 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/11/25 10:31:22 INFO ResourceProfile: Limiting resource is cpu
22/11/25 10:31:22 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/11/25 10:31:23 INFO SecurityManager: Changing view acls to: A
22/11/25 10:31:23 INFO SecurityManager: Changing modify acls to: A
22/11/25 10:31:23 INFO SecurityManager: Changing view acls groups to:
22/11/25 10:31:23 INFO SecurityManager: Changing modify acls groups to:
22/11/25 10:31:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(A); groups with view permissions: Set(); users  with modify permissions: Set(A); groups with modify permissions: Set()
22/11/25 10:31:24 INFO Utils: Successfully started service 'sparkDriver' on port 9768.
22/11/25 10:31:24 INFO SparkEnv: Registering MapOutputTracker
22/11/25 10:31:24 INFO SparkEnv: Registering BlockManagerMaster
22/11/25 10:31:24 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/11/25 10:31:24 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/11/25 10:31:24 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/11/25 10:31:24 INFO DiskBlockManager: Created local directory at C:\Users\A\AppData\Local\Temp\blockmgr-0e3f07cd-29b9-44ce-b88d-73184c8154dd
22/11/25 10:31:24 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
22/11/25 10:31:24 INFO SparkEnv: Registering OutputCommitCoordinator
22/11/25 10:31:24 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/11/25 10:31:24 INFO Executor: Starting executor ID driver on host host.docker.internal
22/11/25 10:31:24 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
22/11/25 10:31:25 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 9783.
22/11/25 10:31:25 INFO NettyBlockTransferService: Server created on host.docker.internal:9783
22/11/25 10:31:25 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/11/25 10:31:25 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, host.docker.internal, 9783, None)
22/11/25 10:31:25 INFO BlockManagerMasterEndpoint: Registering block manager host.docker.internal:9783 with 434.4 MiB RAM, BlockManagerId(driver, host.docker.internal, 9783, None)
22/11/25 10:31:25 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, host.docker.internal, 9783, None)
22/11/25 10:31:25 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, host.docker.internal, 9783, None)
22/11/25 10:31:26 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 221.4 KiB, free 434.2 MiB)
22/11/25 10:31:26 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.5 KiB, free 434.2 MiB)
22/11/25 10:31:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on host.docker.internal:9783 (size: 32.5 KiB, free: 434.4 MiB)
22/11/25 10:31:26 INFO SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:0
Traceback (most recent call last):
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 458, in dumps
    return cloudpickle.dumps(obj, pickle_protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 602, in dump
    return Pickler.dump(self, obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 692, in reducer_override
    return self._function_reduce(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 565, in _function_reduce
    return self._dynamic_function_reduce(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 546, in _dynamic_function_reduce
    state = _function_getstate(func)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 157, in _function_getstate
    f_globals_ref = _extract_code_globals(func.__code__)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py", line 334, in _extract_code_globals
    out_names = {names[oparg]: None for _, oparg in _walk_global_ops(co)}
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py", line 334, in <dictcomp>
    out_names = {names[oparg]: None for _, oparg in _walk_global_ops(co)}
                 ~~~~~^^^^^^^
IndexError: tuple index out of range
Traceback (most recent call last):
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 458, in dumps
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 73, in dumps
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 602, in dump
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 692, in reducer_override
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 565, in _function_reduce
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 546, in _dynamic_function_reduce
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle_fast.py", line 157, in _function_getstate
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py", line 334, in _extract_code_globals
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py", line 334, in <dictcomp>
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\SparkCourse\ratings-counter.py", line 9, in <module>
    result = ratings.countByValue()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\rdd.py", line 1755, in countByValue
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\rdd.py", line 1250, in reduce
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\rdd.py", line 1197, in collect
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\rdd.py", line 3505, in _jrdd
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\rdd.py", line 3362, in _wrap_function
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\rdd.py", line 3345, in _prepare_for_python_RDD
  File "C:\ProgramData\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 468, in dumps
_pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
22/11/25 10:31:26 INFO SparkContext: Invoking stop() from shutdown hook
22/11/25 10:31:26 INFO SparkUI: Stopped Spark web UI at http://host.docker.internal:4040
22/11/25 10:31:26 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/11/25 10:31:26 INFO MemoryStore: MemoryStore cleared
22/11/25 10:31:26 INFO BlockManager: BlockManager stopped
22/11/25 10:31:26 INFO BlockManagerMaster: BlockManagerMaster stopped
22/11/25 10:31:26 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/11/25 10:31:26 INFO SparkContext: Successfully stopped SparkContext
22/11/25 10:31:26 INFO ShutdownHookManager: Shutdown hook called
22/11/25 10:31:26 INFO ShutdownHookManager: Deleting directory C:\Users\A\AppData\Local\Temp\spark-52f9be3d-6110-4d1e-a067-cd0c67ca8195\pyspark-f97e1df3-473a-414a-939e-baf3bf87d08f
22/11/25 10:31:26 INFO ShutdownHookManager: Deleting directory C:\Users\A\AppData\Local\Temp\spark-6a3b37e2-9ab6-4129-a5b1-a10cce43814e
22/11/25 10:31:26 INFO ShutdownHookManager: Deleting directory C:\Users\A\AppData\Local\Temp\spark-52f9be3d-6110-4d1e-a067-cd0c67ca8195

问题根源分析

  1. 序列化兼容问题：核心报错是PySpark自带的cloudpickle库在序列化Lambda函数时触发IndexError，本质是Spark 3.3.1与部分Python版本（如3.10+）存在兼容性冲突，导致函数全局变量提取逻辑异常。
  2. 脚本匹配问题：报错中提到运行的文件是ratings-counter.py，但提供的代码是WordCount任务，可能存在运行脚本与代码逻辑不匹配的情况。

解决方案

  1. 确认运行脚本正确性：确保执行的是目标WordCount脚本，使用命令spark-submit your-wordcount-script.py（替换为实际脚本文件名）。
  2. 替换Lambda为具名函数：避免Lambda函数的序列化问题，改用普通函数实现：
    def split_line(line):
    return line.split()

words = input.flatMap(split_line)
  3. 调整Spark/Python版本：
    • 将Spark版本升级到3.3.2及以上，或降级到3.2.x版本，修复cloudpickle兼容问题。
    • 确保Python版本在Spark 3.3.1支持范围内（官方推荐3.7-3.10），避免使用Python 3.11+。

内容的提问来源于stack exchange，提问作者user20551429

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.08.10 21:10:45