Let's break down exactly what's causing this error and walk through how to fix it step by step.

What's Going Wrong? #

Your create_dataset function uses zip(*word_pairs) to return data meant to be unpacked into source and target. This works only if every entry in word_pairs is a list of exactly 2 elements—meaning every line in your input file splits into two parts when using \t as the separator.

The error ValueError: too many values to unpack (expected 2) tells us at least one line in your dataset breaks this rule:

• Either a line has no \t separator (so split('\t') returns a single element),
• Or a line has multiple \t separators (so split('\t') returns 3+ elements).

When you run zip(*word_pairs) on such data, it creates more than two iterators, which can't be unpacked into just source and target.

Step-by-Step Fixes #

1. First, Debug to Find Problematic Lines

Let's modify your function to log exactly which lines are breaking the expected format:

def create_dataset(path, num_examples):
    lines = io.open(path, encoding='UTF-8').read().strip().split('\n')
    word_pairs = []
    for idx, line in enumerate(lines[:num_examples]):
        parts = line.split('\t')
        # Check if the line splits into exactly 2 parts
        if len(parts) != 2:
            print(f"⚠️ Problematic line at index {idx}: {repr(line)}")
            print(f"Split into {len(parts)} parts: {parts}\n")
        processed_parts = [preprocess_sentence(w) for w in parts]
        word_pairs.append(processed_parts)
    print(f"Loaded data from {path}")
    return zip(*word_pairs)

Run this, and you'll see exactly which lines are causing issues. This helps you decide whether to fix the source data or adjust your code.

2. Clean the Data or Handle Abnormal Lines

You have two main options here:

• Fix the source data: Manually edit your dataset file to remove extra \t characters or add missing separators in problematic lines.
• Adjust the code to handle bad lines: Filter out or normalize lines that don't fit the 2-part format. Here's a robust version of your function that does this:

def create_dataset(path, num_examples):
    lines = io.open(path, encoding='UTF-8').read().strip().split('\n')
    word_pairs = []
    for line in lines[:num_examples]:
        parts = line.split('\t')
        # Only keep lines that split into exactly 2 parts
        if len(parts) == 2:
            processed_pair = [preprocess_sentence(parts[0]), preprocess_sentence(parts[1])]
            word_pairs.append(processed_pair)
        # Optional: Handle lines with missing parts (uncomment if needed)
        # elif len(parts) == 1:
        #     word_pairs.append([preprocess_sentence(parts[0]), ""])
    
    # Convert zip output to tuples (since zip returns an iterator that can only be used once)
    if not word_pairs:
        return (), ()  # Return empty tuples if no valid pairs found
    source, target = zip(*word_pairs)
    return source, target

Note: We convert the zip result to tuples here because zip iterators can only be iterated once—this lets you safely do things like print(source[-1]) without issues.

3. Verify the Fix

After updating the function, test it to confirm everything works:

sample_size=60000
source, target = create_dataset(data_path, sample_size)
print(f"Valid pairs found: {len(source)}")
print(f"Last source sentence: {source[-1]}")
print(f"Last target sentence: {target[-1]}")

Key Takeaway #

This error almost always boils down to mismatched data format. Ensuring every line in your dataset splits into exactly two parts (as your code expects) is the core fix. The debug step helps you pinpoint exactly where the data is off, so you can choose the best solution for your use case.

内容的提问来源于stack exchange，提问作者Namalians