Win10下Hadoop运行WordCount任务遇resource-types.xml缺失且停滞

问题现象 #

在Windows 10系统中运行Hadoop 3.2.1的WordCount任务时，命令行输出resource-types.xml not found的INFO级提示，且任务提交后陷入停滞，无法继续推进。

相关任务与配置信息 #

1. 命令行任务输出 #

PS C:\hadoop-env\hadoop-3.2.1> hadoop jar C:\hadoop-env\hadoop-3.2.1\share\hadoop\mapreduce\hadoop-mapreduce-examples-3.2.1.jar wordcount /input/senbad.txt /output
2023-04-15 14:03:47,011 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2023-04-15 14:03:47,856 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/shekh/.staging/job_1681547354686_0002
2023-04-15 14:03:47,960 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-04-15 14:03:48,095 INFO input.FileInputFormat: Total input files to process : 1
2023-04-15 14:03:48,150 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-04-15 14:03:48,176 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-04-15 14:03:48,189 INFO mapreduce.JobSubmitter: number of splits:1
2023-04-15 14:03:48,286 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-04-15 14:03:48,302 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1681547354686_0002
2023-04-15 14:03:48,303 INFO mapreduce.JobSubmitter: Executing with tokens: []
2023-04-15 14:03:48,498 INFO conf.Configuration: resource-types.xml not found
2023-04-15 14:03:48,499 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-04-15 14:03:48,969 INFO impl.YarnClientImpl: Submitted application application_1681547354686_0002
2023-04-15 14:03:49,014 INFO mapreduce.Job: The url to track the job: http://LAPTOP-SEF3N6DS:8088/proxy/application_1681547354686_0002/
2023-04-15 14:03:49,015 INFO mapreduce.Job: Running job: job_1681547354686_0002

2. 核心配置文件内容 #

core-site.xml

<configuration>
   <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>

hdfs-site.xml

<configuration>
<property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
   <property>
       <name>dfs.namenode.name.dir</name>
       <value>C:\hadoop-env\hadoop-3.2.1\data\dfs\namenode</value>
   </property>
   <property>
       <name>dfs.datanode.data.dir</name>
       <value>C:\hadoop-env\hadoop-3.2.1\data\dfs\datanode</value>
   </property>

</configuration>

mapred-site.xml

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

yarn-site.xml

<configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
   </property>
   <property>
          <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

3. 已尝试的操作 #

在%HADOOP_HOME%\etc\hadoop目录下创建了resource-types.xml，配置内容如下，但问题仍未解决：

<configuration>
  <resources>
    <resourceType name="GPU" units="NONE">
      <schedulerInclude>true</schedulerInclude>
      <yarnInclude>true</yarnInclude>
    </resourceType>
    <resourceType name="FPGA" units="NONE">
      <schedulerInclude>true</schedulerInclude>
      <yarnInclude>true</yarnInclude>
    </resourceType>
  </resources>
</configuration>

且未创建该文件的其他用户可正常运行任务。

解决办法 #

1. 明确resource-types.xml提示的本质 #

该提示为INFO级别，并非任务停滞的直接原因。resource-types.xml用于GPU/FPGA等特殊资源调度，普通WordCount任务完全不需要，可直接忽略或删除该文件。

2. 排查YARN服务状态与配置 #

• 检查服务运行状态：执行jps命令，确认ResourceManager、NodeManager、NameNode、DataNode均正常启动。若NodeManager未运行，手动启动：

  yarn nodemanager
  

  查看C:\hadoop-env\hadoop-3.2.1\logs目录下的yarn-*-nodemanager-*.log日志，定位启动失败原因。
• 调整yarn-site.xml配置：添加Windows环境适配属性，解决资源分配与虚拟内存检查问题：

  <property>
      <name>yarn.nodemanager.resource.memory-mb</name>
      <value>2048</value>
  </property>
  <property>
      <name>yarn.scheduler.minimum-allocation-mb</name>
      <value>512</value>
  </property>
  <property>
      <name>yarn.nodemanager.vmem-check-enabled</name>
      <value>false</value>
  </property>
  

  修改后重启YARN服务：

  yarn stop-all
  yarn start-all
  

3. 验证环境与输入路径 #

• 检查环境变量：确保HADOOP_HOME、JAVA_HOME（推荐JDK8）配置正确，且%HADOOP_HOME%\bin、%HADOOP_HOME%\sbin已加入系统PATH。
• 确认HDFS输入文件：命令中的/input/senbad.txt是HDFS路径，先执行以下命令验证文件存在：

  hdfs dfs -ls /input
  

  若文件不存在，先上传本地文件到HDFS：

  hdfs dfs -put C:\local\path\senbad.txt /input/
  

内容的提问来源于stack exchange，提问作者Shekhar Gangal