NVMe SSD Python吞吐量测试与fio结果差异原因咨询

Great question—this is a super common pitfall when building your own storage benchmarking code, especially compared to a battle-tested tool like fio. Let’s break down exactly why your Python script is showing inflated speeds, and what’s happening under the hood:

1. You’re Measuring Cache Writes, Not Disk Writes #

The biggest culprit here is kernel page caching. Your Python code uses os.write() without bypassing the system’s cache, while your fio job explicitly disables buffering with buffered=0.

• When you call os.write() in your script, the data is first copied to the kernel’s page cache in RAM—this is fast, but it doesn’t mean the data has actually been written to your Samsung 950 Pro. The kernel handles flushing this cache to disk in the background, but your timer stops as soon as the data hits the cache, not the SSD.
• fio’s buffered=0 enables direct I/O (O_DIRECT flag under the hood), which skips the page cache entirely. Every write goes straight to the disk, so fio’s timer includes the actual time it takes for the SSD to process and persist the data.

2. Non-Blocking I/O Is Misused (And Skewing Your Metrics) #

Your script uses os.O_NONBLOCK when opening the file, but you don’t handle the case where os.write() might not write the full block size.

In non-blocking mode, if the kernel’s write buffer is full, os.write() will return fewer bytes than requested (or even EAGAIN). But your code assumes every call writes the entire blocksize, so you’re overcounting the actual bytes written. Dividing your intended num_bytes by the elapsed time gives you a higher speed than what was actually achieved.

3. You’re Not Waiting for Data to Hit the Disk #

Even if you didn’t use direct I/O, your script closes the file immediately after writing and stops the timer. Closing a file triggers a flush of the cache, but it doesn’t wait for that flush to complete.

fio, on the other hand, ensures all I/O operations are fully completed (data is persisted to disk) before stopping its timer. Your Python code’s timer doesn’t include the time it takes for the kernel to sync the cache to the SSD, so you’re only measuring the time to fill the cache, not the actual write throughput to the drive.

4. Minor: Block Alignment and IO Engine Differences #

fio’s libaio engine uses asynchronous I/O, which allows it to queue multiple write requests to the SSD, leveraging the drive’s ability to handle parallel operations. Your Python script uses synchronous, single-threaded writes, but again, the cache makes this seem faster than it would be without buffering.

Additionally, O_DIRECT requires write sizes and offsets to be aligned to the disk’s sector size (usually 512 bytes or 4KB). Your script doesn’t check for this, which would cause errors if you added direct I/O without fixing alignment.

Fixing Your Python Script to Match fio’s Behavior #

To get accurate, comparable results, adjust your script to mimic fio’s key settings:

import os
import time

def perform_timed_write(num_bytes, blocksize, fd):
    """
    Accurate timed write test that matches fio's direct, synced behavior
    """
    # Ensure block size is aligned for O_DIRECT (required on Linux)
    if blocksize % 512 != 0:
        raise ValueError("Block size must be aligned to 512 bytes for direct I/O")
    
    # Generate random data once (mimic fio's write pattern)
    random_byte_string = os.urandom(blocksize)
    
    # Open file with direct I/O (bypass cache) and no non-blocking flag
    write_file = os.open(fd, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
    
    bytes_written = 0
    before_write = time.perf_counter()  # Use perf_counter for high-precision timing
    
    while bytes_written < num_bytes:
        # Handle partial writes (critical for accurate byte count)
        written = os.write(write_file, random_byte_string)
        if written == 0:
            raise IOError("Unexpected end of file during write")
        bytes_written += written
    
    # Force all data to be written to disk before stopping the timer
    os.fsync(write_file)
    after_write = time.perf_counter()
    
    os.close(write_file)
    
    elapsed_time = after_write - before_write
    bytes_per_second = bytes_written / elapsed_time
    return bytes_per_second

With these changes, your Python script will:

• Bypass the page cache (like fio’s buffered=0)
• Wait for data to fully persist to disk (via os.fsync())
• Accurately count actual bytes written
• Use high-precision timing

Final Takeaway #

fio is designed to handle all the low-level nuances of storage benchmarking—cache bypass, I/O alignment, syncing, and error handling—out of the box. Custom scripts often skip these details, leading to misleadingly high results because they’re measuring RAM speed, not SSD speed.

内容的提问来源于stack exchange，提问作者John Frye