循环打印datetime.now()触发Errno 34: Results too large偶发问题求助

Hey there, let's tackle this tricky intermittent error you're seeing. Errno 34 (EFBIG) usually screams "file too big" at first glance, but it’s weird that it’s popping up when you call datetime.now(). Let’s break down why this might be happening and how to fix it.

Possible Causes & Fixes #

1. Hidden File System Operations with Timezone Data #

Most people don’t realize this, but datetime.now() under the hood reads system timezone files (like those in /usr/share/zoneinfo/) to get local time. If those files become corrupted, unreadable, or your process hits a file descriptor limit and can’t open them, you’ll get this unexpected error.

Fix: Pre-fetch your local timezone once, instead of letting datetime.now() read the file every loop iteration:

from datetime import datetime, timezone

# Grab the local timezone ONCE before the loop starts
local_timezone = datetime.now(timezone.utc).astimezone().tzinfo

count = 0
total = 1000000

while count < total:
    start_time = datetime.now(local_timezone)
    # Your core loop logic here
    count += 1
    end_time = datetime.now(local_timezone)
    print(f"Loop started at {start_time}, took {end_time - start_time}")

This cuts down on repeated file I/O and reduces the chance of hitting issues with timezone file access.

2. Resource Limits (Too Many Open File Descriptors) #

If your loop runs for a long time and opens files/sockets without closing them, your process might hit the system’s file descriptor limit. When datetime.now() tries to open the timezone file, it can’t get a new descriptor and throws Errno 34 as a side effect.

Fixes:

• Add file descriptor logging to catch when this happens:

  import os
  import logging
  
  logging.basicConfig(filename='loop_debug.log', level=logging.DEBUG)
  
  # Inside your except block for Errno 34:
  fd_count = len(os.listdir(f'/proc/{os.getpid()}/fd'))
  logging.error(f"Errno 34 hit! Open file descriptors: {fd_count}")
  

• Raise the file descriptor limit (Linux example):
  • Temporarily: Run ulimit -n 4096 before starting your script.
  • Permanently: Edit /etc/security/limits.conf to add your_username soft nofile 4096 and your_username hard nofile 8192, then restart your session.

3. Add Robust Error Handling for Intermittent Failures #

Since the issue is hard to reproduce, you need to catch the error, log context, and recover gracefully so your loop doesn’t crash.

Example Error Handling:

import datetime
import os
import logging
import importlib

logging.basicConfig(filename='loop_errors.log', level=logging.ERROR)

count = 0
total = 1000000

while count < total:
    start_time = None
    try:
        start_time = datetime.datetime.now()
        # Your loop logic here
        count += 1
        end_time = datetime.datetime.now()
        print(f"Loop started at {start_time}, took {end_time - start_time}")
    except OSError as e:
        if e.errno == 34:
            # Log every detail you can
            current_pid = os.getpid()
            fd_count = len(os.listdir(f'/proc/{current_pid}/fd')) if os.path.exists(f'/proc/{current_pid}/fd') else 'Unknown'
            tz_var = os.environ.get('TZ', 'Not set')
            
            logging.error(f"Errno 34 encountered at count={count}, total={total}")
            logging.error(f"Open file descriptors: {fd_count}")
            logging.error(f"TZ environment variable: {tz_var}")
            logging.error(f"Full error message: {str(e)}")
            
            # Try to recover: reload datetime module and use UTC as fallback
            importlib.reload(datetime)
            start_time = datetime.datetime.utcnow()
            end_time = datetime.datetime.utcnow()
            print(f"Recovered! Loop started at UTC {start_time}, took {end_time - start_time}")
        else:
            # Re-raise other errors since they're not what we're handling
            raise

4. Check System-Level Issues #

Occasionally, this error can stem from:

• Corrupted timezone files: Verify the integrity of files in /usr/share/zoneinfo/ (reinstall the tzdata package if needed).
• Low disk space: If your system’s root partition is full, it can cause weird file operation errors. Run df -h to check.

内容的提问来源于stack exchange，提问作者Myco