Python3处理1-10MB文本：按ID分组列表的效率优化咨询

Great question—your current approach works, but those repeated if-elif checks can add up, especially as your file size grows. Let's break down why it's slow and how to fix it.

Why Your Current Code Is Slow #

Each line you read triggers a chain of conditional checks to find the right list. For a 10MB file with thousands of lines, that's a lot of unnecessary comparisons. Plus, having separate variables for each list (list_1, list_2, etc.) is hard to scale if your ID range expands later.

Optimized Solution: Use a Dictionary + CSV Module #

The best fix here is to use a dictionary where keys are your IDs, and values are the lists associated with each ID. This lets you look up the correct list in constant time (O(1)) instead of checking every condition. Additionally, Python's built-in csv module will make parsing your comma-separated data faster and more robust than manually splitting strings.

Option 1: Using defaultdict for Flexible List Creation

This approach automatically creates empty lists for new IDs, so you don't have to predefine every possible list upfront:

from collections import defaultdict

# Initialize a dictionary that generates empty lists for new IDs
id_lists = defaultdict(list)

with open('your_file.txt', 'r', encoding='utf-8') as txt:
    # Skip the header line (remove this if your file doesn't have one)
    next(txt)
    for line in txt:
        parts = line.rstrip().split(',')
        try:
            id_num = int(parts[2])
            id_lists[id_num].append(parts)
        except (IndexError, ValueError):
            # Handle lines with invalid format (optional but recommended)
            print(f"Skipping invalid line: {line.strip()}")

# Access your lists like this:
list_1 = id_lists[1]
list_2 = id_lists[2]
# ... and so on

Option 2: Using the csv Module (More Robust)

If your data is strictly CSV-formatted (even if it has edge cases like commas inside quotes), the csv module is optimized for this and will be faster than manual string splitting:

import csv
from collections import defaultdict

id_lists = defaultdict(list)

with open('your_file.txt', 'r', encoding='utf-8') as txt:
    reader = csv.reader(txt)
    next(reader)  # Skip header line
    for row in reader:
        try:
            id_num = int(row[2])
            id_lists[id_num].append(row)
        except (IndexError, ValueError):
            print(f"Skipping invalid row: {row}")

# Access lists via id_lists[1], id_lists[2], etc.

Why This Is Faster #

• Dictionary lookups: Instead of checking every if-elif condition, we directly map the ID to its list in one step.
• Optimized parsing: The csv.reader is implemented in C, so it handles string splitting much more efficiently than manual split(',') calls.
• Scalability: This approach works seamlessly even if your ID range grows beyond 7 values—no need to add more conditionals.

内容的提问来源于stack exchange，提问作者Nik Konst