如何高效将大型数据集的DataFrame转换为列表？

Your original code works great for small datasets, but with 300k rows and 300 columns, those nested Python loops will grind to a halt—Python-level iteration just doesn’t scale well with that volume of data. Let’s fix this by leaning into Pandas’ optimized vectorized operations and built-in functions, which run under the hood in C for maximum speed.

Key Bottlenecks in the Original Code #

• The nested for loops iterate over every single cell in Python, which translates to 90 million operations (300k * 300) — that’s slow, inefficient, and unnecessary.
• It converts all values to strings, including empty/NaN entries, which you almost certainly don’t want in your transaction lists (Apriori doesn’t care about empty items).

Optimized Solutions #

Solution 1: Fast Row-wise Processing with apply #

This approach uses Pandas’ apply to process rows in bulk, filtering out empty values before converting to lists:

import pandas as pd

# Read CSV, letting Pandas handle empty cells as NaN
dataset = pd.read_csv('datasetFile.csv')

# Convert each row to a list of non-null string items
transactions = dataset.apply(
    lambda row: [str(item) for item in row if pd.notna(item)],
    axis=1
).tolist()

Why this works better:

• apply processes rows in optimized batches instead of slow Python-level loops, giving you a 10-100x speedup depending on your dataset.
• We explicitly filter out NaN values, so your transaction lists only contain actual items (no useless "nan" strings cluttering your data).

Solution 2: Ultra-Efficient Reshaping with stack #

For the absolute best performance with massive datasets, use Pandas’ stack method to reshape the data, then group rows to build transaction lists:

import pandas as pd

dataset = pd.read_csv('datasetFile.csv')

# Reshape the DataFrame: stack columns into rows, keeping original row indices
stacked_data = dataset.stack().reset_index(level=1, drop=True)

# Group by original row index and convert each group to a list
transactions = stacked_data.groupby(level=0).apply(list).tolist()

Why this is the top choice for large data:

• stack is a fully vectorized operation that runs in C, avoiding any Python loops entirely.
• Grouping and aggregating is handled by Pandas’ optimized backend, making this the fastest option for 300k+ rows.
• It automatically drops NaN values (since stack excludes them by default), so no extra filtering is needed.

Extra Tips for Better Performance #

• If your CSV has lots of empty columns, use usecols in pd.read_csv to load only columns with data — this cuts down on memory usage and processing time.
• Skip type conversion later by reading the CSV with dtype=str:

  dataset = pd.read_csv('datasetFile.csv', dtype=str)
  

内容的提问来源于stack exchange，提问作者Jay