在Databricks中导入sklearn为何会替换Python根日志器？

一、import sklearn时__init__.py执行完毕后发生的流程 #

• 模块导入收尾阶段：Python解释器会执行模块的后续初始化钩子，包括__spec__相关处理、sys.modules缓存更新，还可能触发第三方库注册的导入回调（如setuptools入口点、numpy/distutils内部钩子）
• numpy.distutils的潜在操作：sklearn依赖numpy，而numpy.distutils可能在自身导入完成后，通过导入副作用修改全局logging配置。尤其是Databricks runtime环境下，numpy的初始化逻辑可能被定制，会替换logging.root为自身的Log类
• 延迟导入的副作用：sklearn导入时会间接触发numpy子模块的延迟导入，这些子模块可能在sklearn/__init__.py执行完毕后才完成初始化，进而修改全局logging对象

二、调试方案 #

1. 追踪logging.root的修改时机 #

• 在导入sklearn前后打印logging.root的类名，同时用sys.settrace()设置全局追踪函数，监控对logging.root的赋值操作：

import sys
import logging

def trace_func(frame, event, arg):
    if event == 'assign':
        for name in frame.f_locals:
            if name == 'root' and frame.f_globals.get('__name__') == 'logging':
                print(f"Logging root assigned at: {frame.f_code.co_filename}:{frame.f_lineno}")
                print(f"New value class: {type(frame.f_locals['root'])}")
    return trace_func

sys.settrace(trace_func)

# 检查初始状态
print(f"Initial logging.root class: {type(logging.root)}")
# 导入sklearn
import sklearn
# 检查修改后的状态
print(f"After import sklearn, logging.root class: {type(logging.root)}")

• 导入sklearn前调用logging.basicConfig强制初始化logging，验证是否能阻止替换：

import logging
logging.basicConfig(level=logging.INFO)
print(f"Before sklearn: {type(logging.root)}")
import sklearn
print(f"After sklearn: {type(logging.root)}")

2. 排查numpy.distutils的初始化逻辑 #

• 直接导入numpy.distutils.log，观察是否会修改logging.root：

import logging
print(f"Before numpy.distutils.log: {type(logging.root)}")
from numpy.distutils import log
print(f"After numpy.distutils.log: {type(logging.root)}")

• 查看numpy.distutils.log的源码，定位替换logging.root的代码段——通常是通过logging.setLoggerClass或直接赋值logging.root实现

3. 利用Databricks环境特性调试 #

• 在Notebook中执行%sh pip show numpy sklearn，确认安装版本和路径，对比本地环境的差异
• 导入sklearn前临时修改numpy.distutils.log的行为，阻止其替换logging.root：

import logging
original_setLoggerClass = logging.setLoggerClass

def monkey_patch_setLoggerClass(cls):
    if cls.__name__ == 'Log':
        print(f"Blocked attempt to set logger class to {cls.__name__}")
        return
    original_setLoggerClass(cls)

logging.setLoggerClass = monkey_patch_setLoggerClass

import sklearn
# 恢复原函数
logging.setLoggerClass = original_setLoggerClass
print(f"Final logging.root class: {type(logging.root)}")

三、临时修复方案 #

导入sklearn前保存原始的logging.root和logging.setLoggerClass，导入后恢复：

import logging
# 保存原始状态
original_root = logging.root
original_setLoggerClass = logging.setLoggerClass

# 导入sklearn
import sklearn

# 恢复原始logging配置
logging.root = original_root
logging.setLoggerClass = original_setLoggerClass
# 重新初始化logging
logging.basicConfig(level=logging.INFO)

内容的提问来源于stack exchange，提问作者fonini