Windows+PyCharm环境Spark运行报错求助：Python Worker连接超时等问题

环境信息 #

spark-2.1.0-bin-hadoop2.7.tar.gz
hadoop-2.7.3.tar.gz
scala-2.12.6
PyCharm 2017.1.3
Anaconda3
Windows 8.1

已完成配置 #

• 安装/解压JAVA、SCALA、SPARK、HADOOP并配置环境变量
• 为Windows X64将winutils及hadoop库添加至D:\hadoop-2.7.3\bin
• 将D:\spark-2.1.0-bin-hadoop2.7\python\pyspark复制到D:\Program Files (x86)\Anaconda3\Lib\site-packages
• 配置PyCharm

报错详情 #

运行Spark示例代码时出现两类连锁错误：

1. TypeError：namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'
2. SparkException：Python worker did not connect back in time

完整报错栈：

"D:\Program Files (x86)\Anaconda3\envs\my_new_env_python35\python.exe" "D:/pyProject/spark session/run-tests.py"
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
  File "D:\Program Files (x86)\Anaconda3\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "D:\Program Files (x86)\Anaconda3\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "", line 961, in _find_and_load
  File "", line 950, in _find_and_load_unlocked
  File "", line 646, in _load_unlocked
  File "", line 616, in _load_backward_compatible
  File "D:\spark-2.1.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\__init__.py", line 44, in 
  File "", line 961, in _find_and_load
  File "", line 950, in _find_and_load_unlocked
  File "", line 646, in _load_unlocked
  File "", line 616, in _load_backward_compatible
  File "D:\spark-2.1.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\context.py", line 36, in 
  File "", line 961, in _find_and_load
  File "", line 950, in _find_and_load_unlocked
  File "", line 646, in _load_unlocked
  File "", line 616, in _load_backward_compatible
  File "D:\spark-2.1.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\java_gateway.py", line 25, in 
  File "D:\Program Files (x86)\Anaconda3\lib\platform.py", line 886, in "system node release version machine processor")
  File "D:\spark-2.1.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\serializers.py", line 393, in namedtuple
TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'
[Stage 0:><br> (0 + 2) / 2]18/05/29 08:59:20 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: Python worker did not connect back in time
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:138)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:67)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:99)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Accept timed out
    at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:133)
    ... 12 more
18/05/29 08:59:20 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor

问题根源 #

这俩错误是连锁反应：

• Spark 2.1.0发布时Python 3.6还没普及，PySpark的serializers.py里自定义的namedtuple函数没适配Python 3.6+的API变化（新版本collections.namedtuple新增了module参数，且要求关键字传参），导致启动时直接抛出TypeError。
• 这个错误会中断Python Worker的初始化流程，进而引发Spark的连接超时异常。

解决方案 #

方案1：降级Python版本到3.5（推荐） #

Spark 2.1.0官方支持的Python版本是2.7或3.5，你已经创建了Python 3.5的虚拟环境my_new_env_python35，只需确保：

1. 在PyCharm中切换到该虚拟环境：
   • 打开File > Settings > Project: [你的项目名] > Project Interpreter
   • 点击右上角齿轮，选择Add，找到my_new_env_python35环境的python.exe路径，添加并设为默认解释器。
2. 配置系统环境变量：
   • 添加PYSPARK_PYTHON，值为D:\Program Files (x86)\Anaconda3\envs\my_new_env_python35\python.exe
   • 添加PYSPARK_DRIVER_PYTHON，值同上
   • 确认SPARK_HOME指向D:\spark-2.1.0-bin-hadoop2.7

方案2：修改PySpark的serializers.py文件（适配Python 3.6+） #

如果不想降级Python，可以修改PySpark代码兼容新版本：

1. 找到你复制到Anaconda的pyspark目录：D:\Program Files (x86)\Anaconda3\Lib\site-packages\pyspark
2. 打开serializers.py，找到namedtuple函数（大概在第393行）：
   原代码：

   def namedtuple(typename, field_names, verbose=False, rename=False):
       return collections.namedtuple(typename, field_names, verbose=verbose, rename=rename)
   

   修改为：

   def namedtuple(typename, field_names, verbose=False, rename=False, module=None):
       return collections.namedtuple(typename, field_names, verbose=verbose, rename=rename, module=module)
   

3. 保存文件后重启PyCharm，重新运行代码。

额外检查项 #

• 确保winutils.exe版本和Hadoop版本一致（你用的Hadoop 2.7.3，对应winutils也要是2.7.x版本）
• 确认HADOOP_HOME环境变量指向D:\hadoop-2.7.3，且D:\hadoop-2.7.3\bin在系统PATH中

内容的提问来源于stack exchange，提问作者mathews