无需Pandas：Python实现按特定列值筛选CSV数据

Got it, let's work through this problem together. You need to filter a large, unordered CSV to keep only rows where the 'number of students' is over 2000, and you don't want to use Pandas. Python's built-in csv module is perfect here—it's lightweight, efficient, and doesn't require loading the entire file into memory (critical for big datasets).

Full Working Code #

import csv

# Configure file paths and threshold
INPUT_CSV = "your_input_file.csv"
OUTPUT_CSV = "filtered_schools.csv"
MIN_STUDENTS = 2000

# Open input and output files simultaneously
with open(INPUT_CSV, mode='r', newline='', encoding='utf-8') as infile, \
     open(OUTPUT_CSV, mode='w', newline='', encoding='utf-8') as outfile:
    
    # Use DictReader to access columns by header name
    csv_reader = csv.DictReader(infile)
    # Reuse the original header for the output file
    csv_writer = csv.DictWriter(outfile, fieldnames=csv_reader.fieldnames)
    
    # Write the header row first
    csv_writer.writeheader()
    
    # Process each row one at a time
    for row in csv_reader:
        # Clean and convert the student count to integer
        try:
            student_count = int(row['number of students'].strip())
        except ValueError:
            # Skip rows with invalid student count values
            print(f"Skipping invalid row: {row}")
            continue
        
        # Check if the student count meets our threshold
        if student_count > MIN_STUDENTS:
            csv_writer.writerow(row)

print(f"Filter complete! Results saved to {OUTPUT_CSV}")

Breakdown of How This Works #

• DictReader/DictWriter: These classes let us reference columns by their header labels (like row['number of students']) instead of index numbers. This makes the code readable and resilient if the CSV column order ever changes.
• Memory Efficiency: We process rows one by one, so even huge CSV files won't hog your RAM. This is way better than loading the entire file into a list for small machines or massive datasets.
• Error Handling: The try-except block catches rows where the student count isn't a valid integer (like typos or empty values) and skips them, preventing the script from crashing.
• Encoding: Specifying utf-8 ensures we handle any special characters (like accented location names) correctly.
• Adjustable Threshold: The MIN_STUDENTS variable makes it easy to tweak the cutoff later without editing the core logic.

Extra Tips for Edge Cases #

• Custom Delimiters: If your CSV uses tabs, semicolons, or another delimiter, add the delimiter parameter to DictReader and DictWriter (e.g., delimiter=';').
• Quoted Fields: The csv module automatically handles fields wrapped in quotes (even if they contain commas), so you don't have to worry about breaking rows incorrectly.
• Large File Performance: This approach will scale to even the biggest CSV files since it only holds one row in memory at a time.

内容的提问来源于stack exchange，提问作者monchisan