Hadoop Streaming任务在Reduce端合并阶段挂起问题求助

用Python开发了一个Hadoop Streaming数据转换任务，遇到如下问题：当输入文件较大（如70MB）时，任务会在Reduce阶段挂起；将输入文件缩小至700KB时，任务可正常运行。以下是相关日志与计数器信息：

Reducer容器日志 #

2024-08-23 10:08:26,080 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl : finalMerge called with 11 in-memory map-outputs and 0 on-disk map-outputs
2024-08-23 10:08:26,086 INFO [main] org.apache.hadoop.mapred.Merger : Merging 11 sorted segments
2024-08-23 10:08:26,087 INFO [main] org.apache.hadoop.mapred.Merger : Down to the last merge-pass, with 10 segments left of total size : 79232287 bytes
2024-08-23 10:08:26,466 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl : Merged 11 segments, 79233409 bytes to disk to satisfy reduce memory limit
2024-08-23 10:08:26,469 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl : Merging 1 files, 13421702 bytes from disk
2024-08-23 10:08:26,472 INFO [main] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl : Merging 0 segments, 0 bytes from memory into reduce
2024-08-23 10:08:26,472 INFO [main] org.apache.hadoop.mapred.Merger : Merging 1 sorted segments
2024-08-23 10:08:26,480 INFO [main] org.apache.hadoop.mapred.Merger : Down to the last merge-pass, with 1 segments left of total size : 79233279 bytes

Application Master日志 #

24/08/05 10:08:51 INFO mapreduce.Job: map 100% reduce 100%
24/08/05 10:29:19 INFO mapreduce.Job: Task Id attempt_XXXXXX, Status: FAILED
AttemptID : attempt_XXXXXX Timed out after 1200 secs

计数器信息 #

Map input records: 703,640 (This is correct)
Map output records: 685,583
Reduce input records : 685,583 (not correct)
Custom Counter From Code-ReduceInputRecords :  685,489 (this is counted in code)

可以看到Reduce端实际接收的记录数为685489，与Hadoop统计的685583不符，推测代码卡在sys.stdin相关逻辑处，恳请帮忙分析原因。

内容的提问来源于stack exchange，提问作者Shellong