使用pandas read_csv()读取数据时丢失秒级时间信息的问题

Hey there! Let's troubleshoot why your second-level timestamp data is going missing when reading your CSV with pandas. I’ve dealt with this exact frustration before, so here are some straightforward fixes to try out:

1. Double-Check Your Custom Parsing Function #

Chances are, the issue is in the format specifier you’re using in your custom function. If you’re accidentally omitting the seconds component (:%S), pandas will drop that info entirely.

For example, if your timestamp string looks like 2024-06-15 09:22:37, your function should explicitly include the seconds in the format:

def parse_timestamp(timestamp_str):
    return pd.to_datetime(timestamp_str, format="%Y-%m-%d %H:%M:%S")

If your original function used something like %Y-%m-%d %H:%M (without :%S), that’s exactly why seconds are disappearing.

2. Use Pandas’ Built-in CSV Parsing Tools (No Custom Function Needed) #

You might not even need a custom function—pandas has robust built-in support for timestamps with seconds. Try these approaches instead:

• Auto-infer the timestamp format (works for most standard timestamp strings):

  import pandas as pd
  df = pd.read_csv("your_dataset.csv", parse_dates=["timestamp"])
  

• Specify the exact format (more reliable for non-standard or consistent formats):

  df = pd.read_csv(
      "your_dataset.csv",
      parse_dates=["timestamp"],
      date_parser=lambda x: pd.to_datetime(x, format="%Y-%m-%d %H:%M:%S")
  )
  

• For newer pandas versions, you can directly pass the format in the parse_dates parameter:

  df = pd.read_csv(
      "your_dataset.csv",
      parse_dates={"timestamp": ["timestamp"]},
      date_format="%Y-%m-%d %H:%M:%S"
  )
  

3. Confirm Seconds Are Actually Stored (Not Just Hidden in Display) #

Sometimes seconds are present in the datetime object but don’t show up when you print the DataFrame. To verify:

• Check the data type of the timestamp column (it should be datetime64[ns]):

  print(df["timestamp"].dtype)
  

• Extract the seconds explicitly to confirm they exist:

  print(df["timestamp"].dt.second.head())
  

• If you want seconds to display by default, adjust pandas’ display settings:

  pd.options.display.date_format = "%Y-%m-%d %H:%M:%S"
  print(df)
  

内容的提问来源于stack exchange，提问作者alwaysaskingquestions