按行合并DataFrame列表触发InvalidIndexError的解决咨询

Here are practical steps to resolve the error you're facing, since resetting indexes and using ignore_index=True didn't fix the issue:

1. Check for Duplicate Column Names #

Even if you assume all DataFrames have identical columns, duplicate column names in individual DataFrames can trigger this error. Run this to verify:

for idx, df in enumerate(df_l):
    if df.columns.duplicated().any():
        print(f"DataFrame {idx} has duplicate columns: {df.columns[df.columns.duplicated()]}")

If duplicates exist, rename columns to ensure uniqueness before concatenation.

2. Verify Exact Column Matching (Including Case Sensitivity) #

Pandas treats column names as case-sensitive. A subtle case difference (e.g., UserID vs userid) can cause alignment issues. Check this with:

reference_cols = df_l[0].columns
for idx, df in enumerate(df_l[1:], 1):
    if not df.columns.equals(reference_cols):
        print(f"DataFrame {idx} columns mismatch:")
        print(f"Expected: {reference_cols}")
        print(f"Got: {df.columns}")

Fix any mismatched column names to match the reference set.

3. Ensure Index Uniqueness Post-Reset #

Double-check that resetting indexes actually produced unique indexes for each DataFrame:

for idx, df in enumerate(new_l):
    if not df.index.is_unique:
        print(f"DataFrame {idx} still has non-unique index after reset")

If any do, try creating a new RangeIndex manually instead:

new_l = [df.set_index(pd.RangeIndex(len(df))) for df in df_l]

4. Concatenate in Batches #

Concatenating 200 DataFrames at once might hit edge cases in pandas 1.4.2. Try splitting the list into smaller batches:

def batch_concat(df_list, batch_size=20):
    # Split into smaller batches
    batches = [df_list[i:i+batch_size] for i in range(0, len(df_list), batch_size)]
    # Concatenate each batch first
    batch_results = [pd.concat(batch, ignore_index=True) for batch in batches]
    # Combine batch results
    return pd.concat(batch_results, ignore_index=True)

final_df = batch_concat(df_l)

5. Force Skip Index Alignment #

Explicitly disable index alignment by combining ignore_index=True with join='outer' (the default) and copy=False to avoid unnecessary data copies:

final_df = pd.concat(df_l, axis=0, ignore_index=True, join='outer', copy=False)

If none of these work, isolate the problem by concatenating subsets of the DataFrames (e.g., first 10, then 20) to identify which specific DataFrame(s) are causing the error. This will let you inspect the problematic data in detail.

内容的提问来源于stack exchange，提问作者Maeaex1