如何选用开源格式保存Pandas的DateTimeIndex与多级索引数据？

Great question—dealing with DateTimeIndex and multi-index data in CSV is such a repetitive, annoying hassle, right? CSV just doesn't preserve that crucial metadata like index types or hierarchical structures, forcing you to reset, rename, reparse, and reindex every single time.

Luckily, there are several excellent open-source formats that solve this problem perfectly, no proprietary software required. Here are your top options, along with quick pandas usage examples:

1. Parquet #

• Why it's great: A widely adopted, open-source columnar storage format that natively preserves pandas index structures (including DateTimeIndex and multi-indexes). It's supported across most data tools (Spark, Dask, Python, R, etc.), making it ideal for cross-tool workflows.
• Pandas usage:
  Save your data:

  df.to_parquet('your_data.parquet')
  

  Read it back (no index fixes needed!):

  import pandas as pd
  df = pd.read_parquet('your_data.parquet')
  

• Bonus perks: Excellent compression (smaller file sizes than CSV), fast read/write speeds, and support for complex data types.

2. Feather #

• Why it's great: Built specifically for fast, lightweight exchange between pandas and R, this open-source format is optimized for speed and simplicity. It perfectly retains all index metadata, so your DateTimeIndex or multi-index comes back exactly as you saved it.
• Pandas usage:
  Save:

  df.to_feather('your_data.feather')
  

  Read:

  df = pd.read_feather('your_data.feather')
  

• Bonus perks: Blazing-fast read/write times (faster than Parquet for small-to-medium datasets), and files are easy to share between Python and R users.

3. HDF5 (via pandas' HDFStore) #

• Why it's great: A mature, open-source hierarchical data format that pandas integrates with via HDFStore. It fully preserves index types and structures, and works well for large, single-machine datasets.
• Pandas usage:
  First, install the required dependency:

  pip install pytables
  

  Then save and read:

  # Save
  df.to_hdf('your_data.h5', key='dataset', mode='w')
  # Read back
  df = pd.read_hdf('your_data.h5', key='dataset')
  

• Note: HDF5 is less universal than Parquet (not as well-supported outside of Python/R), but it's a solid choice if you're working exclusively with pandas on a single machine.

Quick Tip #

All these formats eliminate the need for manual index resets or date parsing. When you save your DataFrame, they automatically store index names, types, and hierarchical structure—so when you load it back, it's exactly the same as when you saved it.

内容的提问来源于stack exchange，提问作者Jesse Blocher