升级DBR15.2/Python3.11后日志文件读取导出时丢失

问题现象 #

• 原DBR 10.4/Python 3.8环境下日志记录+Blob导出功能正常，升级到DBR 15.2/Python 3.11后，日志记录功能正常，但导出日志时抛出错误：

FileNotFoundError: [Errno 2] No such file or directory

• 首次导出正常，后续日志文件对Python不可见
• 指定完整日志路径后问题仍存在：报错时执行os.listdir(logpath)看不到目标日志，但%sh ls -als logpath能看到该文件，再次执行os.listdir(logpath)文件又出现

怀疑的代码片段 #

with open(full_path, "r+b") as data:
        blob.upload_blob(data)

日志记录器定义代码 #

import logging
from logging.handlers import RotatingFileHandler
from logging import FileHandler
from datetime import datetime

def define_logger(file_name, debug=False, clear_handlers=False, stream_level_debug=False, info=True):
    """
    :param file_name: Name of the .log file
    :param debug: If true, creates debug level .log file
    :param clear_handlers: to clear the previous log file handlers.
    :param stream_level_debug: receive the log debug level on console.
    :param info: If true, creates info level log file.

        Defines the logger and arranges the handlers based on the parameters.
    :return: logger instance

    Enables log functionality.
    """
    #file_name = os.path.join(os.getcwd(), file_name)
    print(file_name)
    # define a logger
    logger = logging.getLogger(__name__)
    # the level should be the lowest level set in handlers
    logger.setLevel(logging.DEBUG)
    # This is for removing all the handler from the logger instance
    # First time you are running this you don't need it, when you want to define new logger specify
    if clear_handlers is True:
        logger.handlers.clear()
        while len(logger.handlers) > 0:
            logger.removeHandler(logger.handlers[0])

    # Format can be arranged
    # datefmt for formatting the time.
    log_format = logging.Formatter('[%(levelname)s] - %(asctime)s - %(filename)s - %(funcName)s - %(message)s',
     datefmt='%Y-%m-%d %H:%M')

    # For logging into console
    stream_handler = logging.StreamHandler()
    stream_handler.setFormatter(log_format)
    if not stream_level_debug:
        stream_handler.setLevel(logging.INFO)
    else:
        stream_handler.setLevel(logging.DEBUG)
    logger.addHandler(stream_handler)

    # For logging into Info.log
    if info:
        info_handler = RotatingFileHandler('{}_info.log'.format(file_name), backupCount=1)
        info_handler.setFormatter(log_format)
        info_handler.setLevel(logging.INFO)
        logger.addHandler(info_handler)

    # If cfg['log_debug'] = True then it will also capture debug level logs and puts it into _debug.log file
    if debug:
        debug_handler = RotatingFileHandler('{}_debug.log'.format(file_name), backupCount=1)
        debug_handler.setFormatter(log_format)
        debug_handler.setLevel(logging.DEBUG)
        logger.addHandler(debug_handler)

    return logger


# main
mainlog = str(datetime.now().strftime("Dataprep_%Y-%B-%d-at%H-%M"))
log = define_logger(mainlog, debug=cfg['log_debug'], clear_handlers=True, stream_level_debug=cfg["stream_level_debug"])
log.info('=' * 80)
blob_outpath = cfg['blob_outpath']
log.debug('blob_outpath {}'.format(blob_outpath))
# Upload the log to blob drive
log2blob(env, mainlog, blob_outpath, cfg['log_debug']) # This works

解决思路与方案 #

1. 释放日志文件句柄
   Python 3.11对文件资源管理更严格，RotatingFileHandler轮换后可能未完全释放句柄，导致Python进程无法感知文件变化。上传前临时关闭日志处理器，完成后重新初始化：

# 上传前关闭所有文件日志处理器
for handler in log.handlers:
    if isinstance(handler, RotatingFileHandler):
        handler.close()
        log.removeHandler(handler)
# 执行Blob上传
log2blob(env, mainlog, blob_outpath, cfg['log_debug'])
# 重新初始化日志处理器
log = define_logger(mainlog, debug=cfg['log_debug'], clear_handlers=True, stream_level_debug=cfg["stream_level_debug"])

2. 统一使用绝对路径
   DBR升级后默认工作目录可能变化，避免相对路径导致的不一致：

import os
# 定义全局日志目录（使用DBFS挂载点或共享目录）
LOG_ROOT = "/dbfs/mnt/shared_logs"
os.makedirs(LOG_ROOT, exist_ok=True)

# 修改define_logger中的文件路径
info_handler = RotatingFileHandler(os.path.join(LOG_ROOT, f"{file_name}_info.log"), backupCount=1)
debug_handler = RotatingFileHandler(os.path.join(LOG_ROOT, f"{file_name}_debug.log"), backupCount=1)

# 确保log2blob中使用相同路径拼接full_path
full_path = os.path.join(LOG_ROOT, f"{mainlog}_info.log")

3. 修正文件读取模式
   r+b模式要求文件存在且支持读写，上传仅需读权限，同时处理轮换后的备份文件：

import os
def upload_log_to_blob(full_path, blob):
    # 先检查原文件
    if os.path.exists(full_path):
        with open(full_path, "rb") as data:
            blob.upload_blob(data)
    # 原文件不存在则检查轮换后的备份（如.log.1）
    else:
        backup_path = f"{full_path}.1"
        if os.path.exists(backup_path):
            with open(backup_path, "rb") as data:
                blob.upload_blob(data)

4. 绕过Python文件系统缓存
   os.listdir可能存在缓存，改用pathlib直接遍历文件系统：

from pathlib import Path
log_dir = Path(logpath)
# 实时获取目录下的文件
log_files = [f for f in log_dir.iterdir() if f.suffix == ".log"]

内容的提问来源于stack exchange，提问作者ArniBandyo