Hadoop/Spark配置求助：PySpark运行报错及节点异常问题

我是Hadoop和Spark的新手，当前使用PySpark Jupyter环境。已完成环境配置与环境变量设置，但执行代码时持续出现报错：

ERROR datanode.DataNode: BlockSender.sendChunks() exception:
java.io.IOException: An established connection was aborted by the software in your host machine

此外，尝试将PySpark数据保存至HDFS或本地时，进程一直处于执行状态；Resource Manager中显示存在不健康节点。恳请各位协助解决该问题，谢谢。

配置文件信息 #

core-site.xml #

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml #

<configuration>
<property>
   <name>dfs.replication</name>
   <value>1</value>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>file:///C:/hadoop/data/namenode</value>
</property>
<property>
   <name>dfs.datanode.data.dir</name>
   <value>file:///C:/hadoop/data/datanode</value>
</property>
<property>
  <name>dfs.namenode.rpc-address</name>
  <value>localhost:9000</value>
</property>
</configuration>

yarn-site.xml #

<configuration>
<property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
 </property>
 
 <property>
      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>
 
 <property>
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
  <value>98.5</value>
</property>
</configuration>

hadoop-env.cmd 中的JAVA_HOME设置 #

set JAVA_HOME=C:\Java\jdk-1.8

环境变量信息 #

用户变量 #

• JAVA_HOME: C:\Java\jdk-1.8
• HADOOP_HOME: C:\hadoop
• SPARK_HOME: C:\Users\HP\AppData\Local\Programs\Python\Python310\Lib\site-packages\pyspark

路径中已添加C:\spark\spark-2.3.1-bin-hadoop2.7、C:\Java\jdk-1.8和C:\hadoop

系统变量 #

• HADOOP_HOME: C:\hadoop\bin
• SPARK_HOME: C:\spark\spark-2.3.1-bin-hadoop2.7\bin

路径中已添加Hadoop、Java、Spark的bin目录及Hadoop的sbin目录。

内容的提问来源于stack exchange，提问作者Hind Benali