Python大存储磁盘递归扫描效率优化问询

初始场景与代码 #

我尝试通过递归扫描目录来获取磁盘占用大小及每个文件和文件夹的详情，初始代码可正常运行但效率不足，无法适配200GB以上存储的扫描需求：

import os
import sys
import logging
from os.path import *
import datetime  # 补充原代码缺失的datetime导入

def scanSlot(path):
    """Return total size of files in given path and subdirs."""
    global dir_list
    global path_list
    try:
        dir_list = os.scandir(path)
    except Exception as e:
        logging.info(">>> Access Denied for " + path)
        dir_list = {}
    tot_size = 0
    for file in dir_list:
        file_stat = file.stat()
        time_stat = os.stat(file.path)
        # 如果是目录则递归调用
        if file.is_dir(follow_symlinks=False):
            bytes = scanSlot(file.path)
            tot_size += bytes
            path_list.append((file.name,file.path,file_stat.st_mode,file_stat.st_ino,file_stat.st_dev,file_stat.st_nlink,file_stat.st_uid,file_stat.st_gid,tot_size, datetime.fromtimestamp(time_stat.st_atime),datetime.fromtimestamp(time_stat.st_mtime),datetime.fromtimestamp(time_stat.st_ctime),"dir"))
        # 如果是文件则提取详情
        if file.is_file(follow_symlinks=False):
            tot_size += file.stat(follow_symlinks=False).st_size
            path_list.append((file.name,file.path,file_stat.st_mode,file_stat.st_ino,file_stat.st_dev,file_stat.st_nlink,file_stat.st_uid,file_stat.st_gid,file_stat.st_size, datetime.fromtimestamp(time_stat.st_atime),datetime.fromtimestamp(time_stat.st_mtime),datetime.fromtimestamp(time_stat.st_ctime),"file"))
    return tot_size

函数调用示例：

server_size = scanSlot('D:\\New folder')

测试结果 #

• 扫描5.49GB磁盘（含244,169个文件、34,253个文件夹）时，无列表追加操作耗时约8分钟；
• 添加列表追加后耗时约25分钟，列表追加操作是性能瓶颈。

已尝试的优化方案 #

• 使用numba库：因numba未实现os库相关功能，无法生效；
• 考虑转为Cython：暂不确定是否有效；

注：path_list中的详情需保留以用于后续分析，无法省略列表追加操作。

更新优化进展 #

根据建议改用os.walk()实现目录扫描，效率大幅提升：扫描19.6GB磁盘（含275,559个文件、38,592个文件夹）并对每个文件/目录进行日志IO操作，耗时约20分钟。优化后代码如下（仍在测试中）：

import os
import logging
import datetime

def scanSlot(path):
    total_size = 0
    global path_list
    for dirpath, dirnames, filenames in os.walk(path):
        for f in filenames:
            fp = os.path.join(dirpath, f)
            # 跳过符号链接
            if not os.path.islink(fp):
                file_stat = os.stat(fp)
                file_size = os.path.getsize(fp)
                path_list.append((f,fp,file_stat.st_mode,file_stat.st_ino,file_stat.st_dev,file_stat.st_nlink,file_stat.st_uid,file_stat.st_gid,file_size, datetime.fromtimestamp(file_stat.st_atime),datetime.fromtimestamp(file_stat.st_mtime),datetime.fromtimestamp(file_stat.st_ctime),"file"))
                total_size += file_size
    return total_size

if __name__ == "__main__":
    path_list = []  # 补充全局变量初始化
    logging.info('>>> Start:' + str(datetime.now().time()))
    # 遍历根目录下的所有子目录并计算大小
    for dirpath, dirnames, filenames in os.walk('F:\\'):
        for f in dirnames:
            fp = os.path.join(dirpath, f)
            dir_size = scanSlot(fp)
            file_stat = os.stat(fp)
            logging.info('Dirname:'+str(fp)+'Size:'+str(dir_size))
            path_list.append((f,fp,file_stat.st_mode,file_stat.st_ino,file_stat.st_dev,file_stat.st_nlink,file_stat.st_uid,dir_size, datetime.fromtimestamp(file_stat.st_atime),datetime.fromtimestamp(file_stat.st_mtime),datetime.fromtimestamp(file_stat.st_ctime),"dir"))
    logging.info('>>> End:' + str(datetime.now().time()))

进一步问询 #

如何进一步提升该扫描方案的效率？初步考虑采用multiprocessing模块并行扫描多个磁盘，是否可行？

参考技术点：os.scandir()、is_file、is_dir、logging模块、os.walk()、multiprocessing模块；follow_symlinks参数使用说明。

内容的提问来源于stack exchange，提问作者user9865537