如何高效读写1GB文本/CSV文件？C++内存优化与数据结构选型

Great questions—handling large text/CSV files in C++ efficiently boils down to minimizing IO overhead and memory fragmentation, especially when dealing with lots of string operations. Let’s break this down for your two scenarios:

Reading Tips #

• Use Memory Mapping (mmap): This is hands down the fastest way to read large files in C++. It maps the entire file directly into your process's address space, eliminating explicit read calls and reducing kernel-user mode switches. Here's a quick snippet:

#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>

int fd = open("large_file.txt", O_RDONLY);
off_t file_size = lseek(fd, 0, SEEK_END);
char* file_data = (char*)mmap(NULL, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
// Access file_data like a regular char array
// Cleanup when done
munmap(file_data, file_size);
close(fd);

• Optimize Standard Streams: If you prefer std::ifstream, disable C stdio synchronization and untie from std::cout to cut down overhead:

std::ifstream in("large_file.txt");
in.sync_with_stdio(false);
in.tie(nullptr);

• Chunked Reading: If memory mapping isn’t feasible (e.g., limited RAM), read in fixed-size chunks (64KB–1MB) instead of line-by-line to minimize IO syscalls.

Storage Tips #

• Stream Processing (Avoid Full Load): If you don’t need the entire file in memory, process it incrementally—read a chunk/line, handle it, then write output immediately. This keeps your memory footprint tiny.
• Binary Storage for Processed Data: For long-term storage, convert data to a binary format (custom structs or Protocol Buffers). Binary formats are smaller, faster to read/write, and skip repeated text parsing.
• Compression: Use libraries like zlib to compress stored data if disk space is tight—just ensure efficient decompression when reading later.

Let’s start with memory efficiency, then cover the best data structures for your use case.

Memory-Efficient Storage Strategies #

• Ditch vector<vector<string>>: This is a memory hog—each std::string has overhead (pointer, size, capacity), and nested vectors add extra allocator bloat. Instead:
  • Single Buffer + Offset Tracking: Load the CSV into one contiguous char buffer (via mmap or a large std::string), then store only field start/end offsets. For example:

    struct Row {
        // Each pair = start index + length of field in the buffer
        std::vector<std::pair<size_t, size_t>> fields;
    };
    std::vector<Row> csv_data;
    

    No extra string allocations until you need to work with a field’s content.
  • String Interning: If your CSV has many duplicate strings (e.g., category labels), create an intern pool: an std::unordered_map<std::string_view, size_t> that maps unique strings to an index. Store indices instead of full strings to cut memory usage drastically.
  • Custom Memory Pool: Use a single large std::vector<char> as a pool for all strings, then manage offsets within this pool instead of relying on std::string’s per-allocation overhead.

Best Data Structures for Storage & Management #

• std::vector for Row/Field Offsets: For random row access, std::vector is ideal—it’s contiguous in memory, making it cache-friendly (critical for fast processing of large datasets).
• Hash Maps for Indexing: If you need to query rows by specific string fields (e.g., find all rows where "user_id" equals "123"), use std::unordered_map with interned string indices as keys (instead of full std::strings) to save memory:

  // Key: interned index of the string, Value: list of row indices
  std::unordered_map<size_t, std::vector<size_t>> user_id_to_rows;
  

• Stream Processing (No Full Load): If you only need to process the data once (e.g., filter rows, compute aggregates), skip loading the entire CSV. Parse line-by-line, process each row immediately, and discard it afterward—this keeps memory usage to just one row’s worth of fields.

CSV Parsing Tips to Save Memory #

• Parse In-Place in the Mapped Buffer: When using mmap, scan the buffer directly to find commas/newlines and mark field boundaries. Use std::string_view to reference fields without copying them into std::strings.
• Avoid Naive Splitting: Skip std::getline with string streams—manually scan the buffer for delimiters to avoid unnecessary allocations and speed up parsing.

内容的提问来源于stack exchange，提问作者Faraz Qureshi