如何高效将嵌套元组列表转为仅含元组第二个值的Pandas DataFrame

Given your large dataset (121,044 sublists each containing 30 tuples), efficiency is crucial. Here are the fastest approaches to extract the second element of each tuple and build a Pandas DataFrame:

1. Vectorized Numpy Approach (Fastest) #

Numpy's vectorized operations are optimized for speed, making this the best choice for large datasets:

import numpy as np
import pandas as pd

# Assume your nested tuple list is stored in 'data'
# Convert to a 3D numpy array (shape: (121044, 30, 2))
arr = np.array(data)
# Extract the second element from each tuple (axis=2, index 1)
values = arr[:, :, 1]
# Convert directly to DataFrame
df = pd.DataFrame(values)

Why this works: #

• Converting the nested list to a numpy array leverages C-level optimizations for data handling.
• Slicing arr[:, :, 1] efficiently extracts all second tuple elements in a single vectorized step, avoiding Python-level loops.
• The resulting 2D array maps directly to a DataFrame with 121044 rows and 30 columns.

2. Nested List Comprehension (Fast and Simple) #

List comprehensions are faster than explicit for loops with append because they're optimized in Python:

import pandas as pd

# Extract second elements using nested list comprehension
extracted_data = [[tup[1] for tup in sublist] for sublist in data]
# Create DataFrame directly
df = pd.DataFrame(extracted_data)

Why this works: #

• The nested comprehension iterates through each sublist and tuple in a concise, efficient manner.
• Pandas can directly convert the 2D list of extracted values into a DataFrame without additional reshaping.

Avoid These Slow Approaches #

Steer clear of methods that use explicit loops with append operations, as they're significantly slower for large datasets:

# Slow example to avoid!
extracted_data = []
for sublist in data:
    row = []
    for tup in sublist:
        row.append(tup[1])
    extracted_data.append(row)
df = pd.DataFrame(extracted_data)

Notes: #

• If your tuple second elements are of mixed data types, numpy will default to object dtype, but Pandas handles this seamlessly.
• Both methods above ensure that the order of elements is preserved, matching the original structure of your input data.

内容的提问来源于stack exchange，提问作者匿名用户