Python DataFrame转MATLAB Table报错解析及直接读取.pkl文件解决方案咨询

First, Let's Break Down the Error #

The error Dimensions of the key and value must be the same, or the value must be scalar happens because MATLAB's table() function expects each column to be either a scalar or a 1D array of uniform length when converting a Python DataFrame. Your DataFrame contains nested 1xN objects with their own subfields—these aren't simple numeric/string arrays that MATLAB can map directly to Table columns, hence the dimension mismatch.

Two Effective Workarounds (No .mat Middle Step) #

Option 1: Manually Flatten Nested Fields in MATLAB

This approach gives you full control over how nested data is extracted. Here's a step-by-step implementation:

% Load the pickled DataFrame as you did before
fid = py.open("data.pkl");
data = py.pickle.load(fid);
py.close(fid); % Don't forget to close the file handle!

% Get column names from the DataFrame
cols = cell(data.columns);

% Initialize containers for Table variables and names
table_vars = {};
table_names = {};

for i = 1:length(cols)
    col_name = cols{i};
    col_series = data.(col_name);
    
    % Check if the column contains nested objects (not basic types)
    first_element = col_series.iloc(0);
    is_nested = ~py.isinstance(first_element, py.int) && ...
                ~py.isinstance(first_element, py.float) && ...
                ~py.isinstance(first_element, py.str);
    
    if is_nested
        % Get valid subfield names (filter out Python internal attributes)
        sub_fields = cell(py.dir(first_element));
        sub_fields = sub_fields(~cellfun(@(x) startsWith(x, '__'), sub_fields));
        
        % Extract each subfield into a separate Table column
        for j = 1:length(sub_fields)
            sub_field = sub_fields{j};
            % Convert Python series elements to MATLAB cell array
            sub_data = cell(col_series);
            % Extract subfield values
            sub_data = cellfun(@(obj) obj.(sub_field), sub_data, 'UniformOutput', false);
            
            % Convert to appropriate MATLAB type
            if isnumeric(sub_data{1})
                sub_data = cell2mat(sub_data);
            else
                sub_data = string(sub_data);
            end
            
            table_vars{end+1} = sub_data;
            table_names{end+1} = [col_name '_' sub_field];
        end
    else
        % Convert basic type columns directly to MATLAB arrays
        mat_data = cell2mat(cell(col_series));
        if ischar(mat_data)
            mat_data = string(mat_data);
        end
        table_vars{end+1} = mat_data;
        table_names{end+1} = col_name;
    end
end

% Create the final MATLAB Table
T = table(table_vars{:}, 'VariableNames', table_names);

Option 2: Use Pandas' json_normalize (Simpler for JSON-Serializable Nested Data)

If your nested objects are dictionary-like or have a __dict__ attribute (common for many Python objects), you can leverage Python's pandas.json_normalize() directly in MATLAB to flatten the DataFrame first, then convert to a Table:

% Load the pickled DataFrame
fid = py.open("data.pkl");
data = py.pickle.load(fid);
py.close(fid);

% Flatten nested structures using pandas.json_normalize
flattened_df = py.pandas.json_normalize(data);

% Convert the flattened DataFrame to MATLAB Table
T = table(flattened_df);

This method automatically expands nested subfields into columns named like parent_field.child_field, which works seamlessly for most standard nested data structures.

Key Notes #

• Ensure your MATLAB version is R2020b or newer—these versions have improved compatibility with Python Pandas objects.
• For custom Python objects with non-standard attributes, you may need to adjust the subfield filtering logic in Option 1.
• Always remember to close the file handle with py.close(fid) after loading the pickle file to avoid resource leaks.

Content of the question comes from Stack Exchange, question author David K