Python实现含噪声的&分隔数据清洗并转CSV文件求助

Hey there! Let's get your sensor data cleaned up and converted to a proper CSV file. I'll break this down into simple, beginner-friendly steps since you're new to Python.

Step 1: Understand the Data Issues #

Your data has a few key quirks we need to fix first:

• Noise characters (ÿ, þ, ü) at the start of some lines before the opening {
• HTML-encoded & instead of actual & separating key-value pairs
• All data is wrapped in curly braces {...}

Step 2: Full Working Code #

Here's a corrected, commented script that handles all these issues and avoids common beginner syntax errors:

import csv

# Define the exact columns we expect in the final CSV
FIELDNAMES = ["device_id", "temp", "hum", "vcc", "packet_no", "sleep_time"]

# Update these paths to match your actual input/output files
INPUT_FILE = "your_raw_data.txt"
OUTPUT_FILE = "cleaned_sensor_data.csv"

def clean_and_parse_line(line):
    # Remove extra whitespace (like newlines) from the start/end of the line
    line = line.strip()
    
    # Remove leading noise characters if present
    if line.startswith(("ÿ{", "þ{", "ü{")):
        # Slice off the first noise character
        line = line[1:]
    
    # Skip lines that don't follow the {data} format
    if not (line.startswith("{") and line.endswith("}")):
        print(f"Skipping invalid line: {line}")
        return None
    
    # Remove the curly braces to get the raw key-value string
    data_str = line[1:-1]
    
    # Replace HTML-encoded & with regular & so we can split correctly
    data_str = data_str.replace("&", "&")
    
    # Split into individual key-value pairs
    kv_pairs = data_str.split("&")
    
    # Convert pairs into a dictionary for easy CSV writing
    data_dict = {}
    for pair in kv_pairs:
        key, value = pair.split("=")
        data_dict[key.strip()] = value.strip()
    
    # Ensure all expected columns exist (fill empty if missing)
    for field in FIELDNAMES:
        if field not in data_dict:
            data_dict[field] = ""
    
    return data_dict

# Process the input file and write to CSV
with open(INPUT_FILE, "r", encoding="utf-8") as infile, open(OUTPUT_FILE, "w", newline="", encoding="utf-8") as outfile:
    # Set up the CSV writer with our column names
    writer = csv.DictWriter(outfile, fieldnames=FIELDNAMES)
    
    # Write the header row first
    writer.writeheader()
    
    # Process each line one by one
    for line_num, line in enumerate(infile, 1):
        try:
            cleaned_data = clean_and_parse_line(line)
            if cleaned_data:
                writer.writerow(cleaned_data)
        except Exception as e:
            print(f"Error on line {line_num}: {line.strip()}\nDetails: {e}")

print(f"Done! Cleaned data saved to {OUTPUT_FILE}")

Step 3: Key Fixes & Explanations #

Let's go over the parts that likely caused your original errors:

1. Simplified Noise Handling:

   • Instead of messy nested if statements, we use line.startswith(("ÿ{", "þ{", "ü{")) to check all three noise cases in one line.
   • Slicing line[1:] cleanly removes the first noise character without extra conditionals.

2. HTML Entity Fix:

   • Your data uses & (the HTML code for &), so we replace this first before splitting key-value pairs. This was probably causing your split logic to fail earlier.

3. Error Resilience:

   • We add checks for valid line structure to skip bad lines instead of crashing.
   • A try-except block catches unexpected errors and tells you exactly which line failed, making debugging way easier.

4. Proper CSV Formatting:

   • Using csv.DictWriter takes care of all the CSV formatting rules (like handling commas in values) automatically, so you don't have to build the CSV string manually.

Step 4: How to Use This #

1. Replace INPUT_FILE and OUTPUT_FILE with your actual file paths (e.g., "sensor_logs.txt" and "output.csv").
2. Run the script. It will:
   • Read your input file line by line
   • Clean and parse each valid line
   • Write the structured data to a CSV with matching columns

Common Beginner Pitfalls to Avoid #

• Encoding: Using encoding="utf-8" ensures the special noise characters are read correctly (a frequent source of weird errors).
• Indentation: Python relies on indentation to define code blocks—make sure lines inside if, for, and with are indented properly (this is one of the most common syntax mistakes!).
• Missing Fields: The code adds empty strings for any missing columns, so your CSV won't have gaps or misaligned rows.

内容的提问来源于stack exchange，提问作者Ruchi Isaac