Python大批次写入优化疑问：自研批处理还是依赖内置缓冲？

First, let's unpack what you're seeing and whether custom batching is worth implementing:

Why you see periodic line jumps in wc -l #

The non-linear growth you're observing is a combination of Python's user-space buffering and the OS's kernel page cache. When you call f.write() line-by-line:

• Python accumulates data in its own buffer (default size depends on the system, often 4KB or 8KB for disk files) until it's full, then flushes it to the OS's cache.
• The OS doesn't write cached data to disk immediately—it waits until the cache is full, or until a periodic sync (which is why you see jumps every ~5 minutes). Running wc -l reads the file from disk, so it only sees data that's been persisted, not what's still in the OS cache.

To confirm this, you could check the file's size with os.stat() (which reflects the OS cache size) instead of wc -l—you'll see more incremental growth.

Is your current I/O approach optimal? #

No, it's not. Even with built-in buffering, line-by-line writes carry significant overhead: each f.write() call involves Python-level checks and function call overhead, and while Python batches these into syscalls when its buffer is full, reducing the number of write operations further will still improve performance—especially for 30 million records.

Should you implement custom batching? #

Absolutely. Here's why:

• Fewer syscalls: Batching reduces the number of times Python needs to interact with the OS. For example, writing 100k lines in one call instead of 100k separate calls cuts down on syscall overhead drastically.
• Lower CPU overhead: Each f.write() has fixed per-call costs (like buffer checks). Reducing the number of calls frees up CPU cycles that can be used for your GPU-bound processing instead of waiting on I/O.
• Better control: You can align I/O batches with your existing GPU processing batches, making your pipeline more efficient.

What batch size should you use? #

Your proposed 100k records is a great starting point, but here's how to refine it:

• Memory constraints: Since you can comfortably fit 100k records in memory, this is safe. Avoid batches so large that they cause memory fragmentation or compete with your GPU's memory usage (but 100k is unlikely to be an issue here).
• Optimal chunk size: Aim for batches that are 64KB to 1MB in total size (adjust based on your average line length). For example, if each line is 100 bytes, 10k lines = 1MB—this aligns well with OS page sizes and minimizes the number of syscalls without excessive memory usage.
• Test and iterate: Run small benchmarks with batch sizes like 10k, 50k, 100k, and 200k. Measure total runtime to find the sweet spot—you'll likely see diminishing returns beyond a certain size (e.g., 100k vs 200k might not give a big performance boost).

Implementation Example #

Here's how to integrate custom batching with your existing workflow:

class BatchWriter:
    def __init__(self, filename, batch_size=100000):
        self.file = open(filename, 'w', buffering=1024*1024)  # 1MB Python buffer
        self.batch_size = batch_size
        self.current_batch = []

    def add_record(self, record):
        # Assume `record` is a string without a trailing newline
        self.current_batch.append(record)
        if len(self.current_batch) >= self.batch_size:
            self.flush_batch()

    def flush_batch(self):
        if self.current_batch:
            # Join with newlines and write once
            self.file.write("\n".join(self.current_batch) + "\n")
            self.current_batch = []

    def close(self):
        # Flush any remaining records
        self.flush_batch()
        self.file.close()

# Usage
writer = BatchWriter("output.txt")
for record in your_gpu_batch_generator():
    writer.add_record(record)
writer.close()

Additional Tips #

• Set a larger Python buffer: Opening the file with buffering=1024*1024 (1MB) reduces the number of syscalls even further, complementing your custom batching.
• Avoid unnecessary string operations: If your GPU processing already generates lines with trailing newlines, skip the "\n".join() and use self.file.write("".join(self.current_batch)) instead.
• Don't over-flush: Resist the urge to call f.flush() after every batch—it forces the OS to write to disk immediately, which will slow down performance. Let the OS handle disk syncs unless you need durability guarantees.

内容的提问来源于stack exchange，提问作者pilu