管道输出至head时Python脚本出现OSError: [Errno 22]错误求助

执行以下命令，把所有mongodb.log.*日志文件通过cat传给jsonFetcher.py脚本，再将输出管道到head：

cat *mongodb.log.* | python jsonFetcher.py -b 20240101000000 -e 20250101000000 | head

输出末尾出现如下错误，但将输出管道到其他Python脚本时无此问题：

{"t":{"$date":"2024-06-09T02:47:32.053+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.240.198:57790","uuid":"8e485191-5c63-457e-b341-14515b593ef1","connectionId":207933,"connectionCount":773}}
{"t":{"$date":"2024-06-09T02:47:32.084+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn207932","msg":"client metadata","attr":{"remote":"192.168.240.198:57776","client":"conn207932","negotiatedCompressors":[],"doc":{"application":{"name":"MongoDB Automation Agent v13.16.2.8826 (git: 36d72fa13b663f402e1285ab8458536766897530)"},"driver":{"name":"mongo-go-driver","version":"v1.12.0-cloud"},"os":{"type":"linux","architecture":"arm64"},"platform":"go1.21.10"}}}
{"t":{"$date":"2024-06-09T02:47:32.085+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.240.198:57794","uuid":"3d76d2c2-2298-4fc8-80b6-e029d0d1f933","connectionId":207934,"connectionCount":774}}
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
OSError: [Errno 22] Invalid argument

以下是jsonFetcher.py的代码：

#!/usr/bin/env python3

"""
 File: jsonFetcher.py
 Author: Dennis Spera
 Date: 2024-06-24
 Description: Extract json from mongod.log on stdin based on date ranges.
"""

import sys
import json as j
from datetime import datetime
import re as regex
from commandlines import Command as cmd

b = str()
e = str()
start_d_t_fmt = None
end_d_t_fmt = None
max_input_length = 14
min_input_length = 8

def is_json(json):
    try:
        j.loads(json)
    except ValueError:
        return False
    return True

try:
  c = cmd()
  
  try:  
       b = c.get_definition('b') 
       if len(b) > max_input_length:
           sys.stderr.write('beginning timestamp exceeds beginning timesamp length of ' + str(max_input_length))
           sys.exit(0)
       if len(b) < min_input_length:
           sys.stderr.write('beginning timestamp less than beginning timesamp length of ' + str(min_input_length))
           sys.exit(0)           

       try:
          test = int(b) 
       except:
           sys.stderr.write('beginning timestamp is an invalid data format')
           sys.exit(0)      

       padding = max_input_length - len(b)             

       if padding > 0: 
          for i in range(1, padding+1, 1):
              b=b+'0'
  
       e = c.get_definition('e') 
       if len(e) > max_input_length:
           sys.stderr.write('ending timestamp exceeds ending timesamp length of ' + str(max_input_length))
           sys.exit(0)
       if len(b) < min_input_length:
           sys.stderr.write('ending timestamp less than ending timesamp length of ' + str(min_input_length))
           sys.exit(0)           

       try:
          test = int(e) 
       except:
           sys.stderr.write('end timestamp is an invalid data format')
           sys.exit(0)      

       padding = max_input_length - len(e)             

       if padding > 0: 
          for i in range(1, padding+1, 1):
              e=e+'0'
                
  except:
        sys.stderr.write('error parsing input parameters')
        sys.exit(0) 

  for line in sys.stdin:
     if is_json(line):

        d = j.loads(line)
        dateT = d["t"]["$date"]

        match_object = regex.match( r'(\d{4})([-])(\d{2})([-])(\d{2})([T])(\d{2})([:])(\d{2})([:])(\d{2})(.*)', dateT)
        d_t_fmt = datetime.strptime(match_object.group(1)+match_object.group(3)\
                                   +match_object.group(5)\
                                   +match_object.group(7)+match_object.group(9)
                                   +match_object.group(11), '%Y%m%d%H%M%S')  
        
        match_object = regex.match( r'(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})',  b) 

        try:
          start_d_t_fmt = datetime.strptime(match_object.group(1)+match_object.group(2)\
                                                       +match_object.group(3)\
                                                       +match_object.group(4)+match_object.group(5)
                                                       +match_object.group(6), '%Y%m%d%H%M%S')
        except:
                 sys.stderr.write('error converting start date-time to a datatime format')  

        match_object = regex.match( r'(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})',  e) 

        try:
          end_d_t_fmt = datetime.strptime(match_object.group(1)+match_object.group(2)\
                                                       +match_object.group(3)\
                                                       +match_object.group(4)+match_object.group(5)
                                                       +match_object.group(6), '%Y%m%d%H%M%S')
        except:
                 sys.stderr.write('error converting end date-time to a datatime format')  

        if (start_d_t_fmt <= d_t_fmt) and (end_d_t_fmt >= d_t_fmt) and not regex.match(r'^$', line):
            print(line,end="")


  sys.stdin.close()

except OSError as err:
  print('!!! ',err)

问题分析 #

这个错误的本质是管道断裂：head命令在读取到默认10行数据后会立即关闭标准输出管道，而你的Python脚本此时还在尝试向已经关闭的管道写入数据，导致操作系统抛出OSError: [Errno 22] Invalid argument。

你查阅到的“Windows目录命名”相关资料属于场景混淆——这个问题和文件写入完全无关，纯粹是管道通信的异常。

解决方案 #

要解决这个问题，需要在脚本中捕获管道关闭的异常，避免错误信息输出，有两种可行方式：

1. 针对print语句捕获异常
   将输出的print(line,end="")替换为：

   try:
       print(line, end="")
   except BrokenPipeError:
       # 管道已关闭，直接终止脚本
       sys.exit(0)
   

2. 全局捕获BrokenPipeError
   在脚本最外层的异常处理中添加对BrokenPipeError的捕获：

   except BrokenPipeError:
       # 忽略管道断裂错误，静默退出
       sys.exit(0)
   except OSError as err:
       print('!!! ',err)
   

补充说明 #

当管道另一端的进程（此处为head）提前退出时，当前进程写入stdout会收到SIGPIPE信号，Python会将其转换为BrokenPipeError。你的脚本原本未捕获这个错误，最终被Python解释器在清理阶段抛出，就出现了Exception ignored的提示。

内容的提问来源于stack exchange，提问作者Dennis