如何在Python中将多个函数的执行时长记录为Pandas DataFrame

问题需求 #

需要为collect_vege_images()、remove_duplicates()、count_veges_in_multiple_plates()三个函数的多轮迭代执行生成结构化日志DataFrame，每行记录迭代次数、图片数量、函数名、开始时间、结束时间、耗时，目标格式如下：

Iteration   image count   Function            start time       end time   elapse time
1           200           collect_vege_images  11.00           11.10       0.10
            200           remove_duplicates    11.10           11.15       0.5
            100           count_veges_in_multiple_plates  11.16          11.20       0.4
2           300           collect_vege_images   11.21          11.31       0.10
            150           remove_duplicates     11.31          11.35       0.5
            50            count_veges_in_multiple_plates  11.35          11.39       0.4

现有问题 #

当前代码仅能记录单个函数的总耗时，无法分函数记录每步执行数据，未处理多函数顺序执行的时间链，也未正确更新每步的图片数量。

解决方案 #

1. 封装通用计时装饰器 #

用装饰器统一处理函数的计时和数据收集，无需在每个函数中重复编写计时逻辑，适合初学者快速复用：

import time
import pandas as pd
import os
import shutil

# 全局列表，用于收集每一条执行日志记录
log_records = []

def log_execution(func):
    def wrapper(*args, **kwargs):
        start_time = time.process_time()
        # 执行函数并获取当前步骤对应的图片数量
        img_count = func(*args, **kwargs)
        end_time = time.process_time()
        elapse_time = round(end_time - start_time, 2)
        # 将执行数据存入日志列表
        log_records.append({
            "Function": func.__name__,
            "start time": round(start_time, 2),
            "end time": round(end_time, 2),
            "elapse time": elapse_time,
            "image count": img_count
        })
        return img_count
    return wrapper

2. 修改原有函数适配日志收集 #

让每个函数返回当前操作对应的图片数量，确保日志中的image count字段准确：

VEGE_SOURCE = r'\\path to vegetable images'
VEGE_COUNT = 5
VEGE_DST = r'\\path to storing vegetable images'
VEGEID = 'Potato'
CYCLES = 5

@log_execution
def collect_vege_images(source):
    plate = os.listdir(source)
    if not plate:
        print(f"vege source is empty: {source}")
        return 0
    moved_count = 0
    # 取VEGE_COUNT和源文件数的最小值，避免索引越界
    for _ in range(min(VEGE_COUNT, len(plate))):
        vege = plate.pop(0)
        curr_vege = os.path.join(source, vege)
        shutil.move(curr_vege, VEGE_DST)
        moved_count += 1
    return moved_count

@log_execution
def remove_duplicates(dst):
    # 补充你的去重逻辑，此处为示例逻辑：模拟移除10%重复文件
    files = os.listdir(dst)
    duplicate_count = int(len(files)*0.1)
    remaining_count = len(files) - duplicate_count
    print(f"Removed {duplicate_count} duplicates, {remaining_count} images left")
    return remaining_count

@log_execution
def count_veges_in_multiple_plates(plates_path):
    vege_img_count = 0
    for root, dirs, files in os.walk(plates_path):
        vege_img_count += len(files)
    return vege_img_count

3. 执行多轮迭代并构建目标DataFrame #

按顺序执行函数，最后整理成目标格式：

if __name__ == "__main__":
    log_records = []
    
    for iteration in range(1, CYCLES+1):
        print(f"Round # {iteration}")
        # 按业务顺序执行三个函数，自动收集日志
        collect_vege_images(VEGE_SOURCE)
        remove_duplicates(VEGE_DST)
        count_veges_in_multiple_plates(VEGE_DST)
    
    # 将日志列表转为DataFrame
    df = pd.DataFrame(log_records)
    # 为每轮迭代的3个函数记录分配相同的迭代编号
    df["Iteration"] = pd.np.repeat(range(1, CYCLES+1), 3)
    # 调整列顺序匹配目标格式
    df = df[["Iteration", "image count", "Function", "start time", "end time", "elapse time"]]
    # 设置Iteration为索引，实现相同迭代编号合并显示的效果
    df = df.set_index("Iteration")
    
    # 打印最终结果
    print(df.to_string(index=True))
    # 可选：保存为CSV文件
    df.to_csv("vege_log.csv")

关键说明 #

• 装饰器log_execution自动为函数添加计时和日志收集逻辑，无需修改函数核心业务代码
• 每个函数返回当前步骤的图片数量，确保日志数据准确
• 通过pd.np.repeat为每轮迭代的三条记录分配相同的迭代编号，设置索引后自动合并显示，匹配目标格式

输出示例 #

image count                     Function  start time  end time  elapse time
Iteration                                                                            
1                  5           collect_vege_images       22.10      22.11          0.01
1                190             remove_duplicates       22.11      22.11          0.00
1                190  count_veges_in_multiple_plates       22.11      22.11          0.00
2                  5           collect_vege_images       22.11      22.11          0.00
2                185             remove_duplicates       22.11      22.11          0.00
2                185  count_veges_in_multiple_plates       22.11      22.12          0.01

内容的提问来源于stack exchange，提问作者mnm