Python多进程执行statsmodels OLS回归无法存入结果到全局DataFrame求助

问题原因 #

• 多进程内存隔离：Windows系统下multiprocessing默认采用spawn模式启动子进程，每个子进程会独立复制父进程的所有数据副本，子进程中对全局summaryDF的修改仅作用于子进程自身的副本，父进程的summaryDF完全不会被更新，这是结果为空的核心原因。
• 代码缩进错误：代码中summaryDF = summaryDF.append(summOut, ignore_index = True)这行缩进错误，不在processor函数体内，函数执行时不会运行该逻辑。

修复方案 #

摒弃全局变量传递多进程计算结果的逻辑，改为子进程返回计算结果、父进程统一汇总的模式，同时修正缩进问题，修复后代码如下：

import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
from statistics import mean, median
from multiprocessing import Pool
import datetime
import os

# 保留原有createStmt实现，此处为示例可替换为你的原有逻辑
def createStmt(target, features):
    stmt = f"{target} ~ {' + '.join(features)}"
    interceptOut = 1
    return stmt, interceptOut

if __name__ == '__main__':
    # 读取原始数据
    inLoc='C:\\temp\\retailer_cost\\'
    inName='raw_data_v1_2.csv'
    inFile = inLoc + inName
    df=pd.read_csv(inFile)

    combList = [['a','b','c','d','e'],
                ['a','b','c','d',],
                ['a','b','c','e'],
                ['a','b','d','e'],
                ['a','c','d','e'],
                ['b','c','d','e']]

    # 子进程处理函数，直接返回计算得到的结果字典
    def processor(combin):
        date_time = str(datetime.datetime.now().time())
        stmt,interceptOut = createStmt('adjusted_value', combin)
        print(stmt)
        mod = smf.ols(formula=stmt, data=df)
        results = mod.fit()
        modID = str(date_time) + '_' + str(interceptOut)
        avg = mean(list(results.pvalues))
        mdn = median(list(results.pvalues))
        return {'modelID': modID,  'statement': stmt, 'num_vars': str(len(combin)), 
                'BIC': str(results.bic) ,'AIC': str(results.aic) ,'RSQ': str(results.rsquared) ,
                'RSQ_ADJ': str(results.rsquared_adj),'CONDITION': str(results.condition_number),
                'AVG_PVALUE': str(avg),'MEDIAN_PVALUE': str(mdn)}
    
    # 启动多进程，接收所有子进程返回的结果
    pool = Pool()                         
    result_list = pool.map(processor, combList)
    pool.close()
    pool.join()

    # 父进程统一汇总结果导出
    summaryDF = pd.DataFrame(result_list)
    summaryDF.to_csv('c:\\temp\\olsModelOut.csv', index=False)

内容的提问来源于stack exchange，提问作者Dimbo1979