如何在Python 3.6中实现类似Stata的数据集附加信息及数组/DataFrame保存功能？

Hey there! I totally get what you're after—replicating Stata's super handy notes and characteristics features in Python 3.6, so you can stash all kinds of extra info (reminders, data origins, estimation methods, you name it) right with your DataFrame or array, and save everything together. Let's walk through a few reliable ways to pull this off:

Pandas DataFrames (and Series) have a native attrs dictionary that's perfect for storing metadata. It's simple, integrated, and works seamlessly with pickle-based saving/loading.

How to use it: #

• Attach metadata to your DataFrame:

  import pandas as pd
  
  # Create your DataFrame
  df = pd.DataFrame({'user_id': [101, 102, 103], 'session_duration': [120, 95, 150]})
  
  # Add notes/characteristics
  df.attrs['notes'] = "Reminder: Need to validate session_duration outliers before analysis"
  df.attrs['data_generation'] = "Exported from raw server logs via clean_session_data.py on 2024-05-20"
  df.attrs['var_est_methods'] = {"session_duration": "Used trimmed mean for summary stats"}
  

• Save and load with metadata intact:

  # Save to pickle file
  df.to_pickle('session_data_with_meta.pkl')
  
  # Load back later
  loaded_df = pd.read_pickle('session_data_with_meta.pkl')
  
  # Access your metadata
  print(loaded_df.attrs['notes'])
  print(loaded_df.attrs['var_est_methods'])
  

Quick notes: #

• Pickle is Python-specific, so if you need cross-language access, skip to the HDF5 method below.
• Works perfectly in Python 3.6 with Pandas 0.23 or newer (which is widely compatible with 3.6).

If you want more control (like handling both DataFrames and NumPy arrays, or structuring metadata in specific ways), a simple custom class is the way to go.

Example implementation: #

import pandas as pd
import numpy as np
import pickle

class DataWithMetadata:
    def __init__(self, data, notes=None, characteristics=None):
        self.data = data  # Can be DataFrame, NumPy array, or even list
        self.notes = notes if notes else {}  # Store reminders/todos as dict
        self.characteristics = characteristics if characteristics else {}  # Store data specs/methods

# Usage example with a NumPy array
np_array = np.array([[1, 2], [3, 4], [5, 6]])
my_dataset = DataWithMetadata(
    data=np_array,
    notes={"todo": "Check for missing values in row 2", "last_updated": "2024-05-20"},
    characteristics={"data_source": "Lab experiment measurements", "processing_step": "Normalized to 0-1 range"}
)

# Save the whole object
with open('array_with_meta.pkl', 'wb') as f:
    pickle.dump(my_dataset, f)

# Load it back
with open('array_with_meta.pkl', 'rb') as f:
    loaded_dataset = pickle.load(f)

# Access your data and metadata
print(loaded_dataset.notes['todo'])
print(loaded_dataset.data)

If you're working with big datasets or need to access the data from other languages (like R or MATLAB), HDF5 is an excellent choice. It lets you store multiple datasets and attach metadata to each.

How to implement: #

import pandas as pd

# Save DataFrame with metadata to HDF5
with pd.HDFStore('large_data_with_meta.h5') as store:
    store['user_data'] = df
    # Attach metadata to the stored dataset
    store.get_storer('user_data').attrs['notes'] = "Dataset includes only active users (last login < 30 days)"
    store.get_storer('user_data').attrs['generation_script'] = "generate_user_dataset.py"

# Load back and retrieve metadata
with pd.HDFStore('large_data_with_meta.h5') as store:
    loaded_df = store['user_data']
    data_notes = store.get_storer('user_data').attrs['notes']
    print(data_notes)

Quick notes: #

• HDF5 is more storage-efficient for large datasets than pickle.
• Most data analysis languages have libraries to read HDF5 files, so it's great for cross-tool collaboration.

内容的提问来源于stack exchange，提问作者user8682794