如何在Python中按基金分组执行带HAC修正的滚动窗口回归？

问题说明 #

需要对包含410只基金的面板数据，按基金独立执行带Newey-West（HAC,1987）标准误修正的滚动窗口回归。现有自定义回归函数可输出正确的t统计量，但当前代码未按基金分组，导致回归窗口混入其他基金数据，需调整代码实现分组独立计算。

数据结构示例 #

import pandas as pd
import numpy as np
import statsmodels.formula.api as smf

funds = pd.DataFrame({
    "Fund": ["A", "A", "A", "A", "B", "B", "B", "B"], 
    "Excess_Return": [np.NaN, 0.172, 0.0465, 0.039, 0.003995, -0.022139, 0.009518, 0.03233],
    "Regression_Constant": [1,1,1,1,1,1,1,1], 
    "RMRF": [0.0118,0.0557,0.0129,0.0403,0.0118,0.0557,0.0129,0.0403], 
    "SMB": [0.0445,0.1838,-0.1539,-0.0496,0.0445,0.1838,-0.1539,-0.0496], 
    "HML": [-0.0189,-0.0981,0.0823,0.0725,-0.0189,-0.0981,0.0823,0.0725], 
    "RMW": [-0.0629,-0.1876,0.1182,0.0767,-0.0629,-0.1876,0.1182,0.0767], 
    "CMA": [0.0474,-0.0035,-0.0161,0.0562,0.0474,-0.0035,-0.0161,0.0562]})

自定义回归函数 #

min_t = 30  # 回归所需最小有效样本量
t_window = 36  # 滚动窗口大小

def process(window_df):
    if window_df["Excess_Return"].count() >= min_t:
        reg = smf.ols(
            "Excess_Return ~ RMRF + SMB + HML + RMW + CMA", 
            data=window_df
        ).fit(cov_type="HAC", cov_kwds={"maxlags":1})
        return [
            # 回归系数
            reg.params[0], reg.params["RMRF"], reg.params["SMB"],
            reg.params["HML"], reg.params["RMW"], reg.params["CMA"],
            # t统计量
            reg.tvalues[0], reg.tvalues["RMRF"], reg.tvalues["SMB"],
            reg.tvalues["HML"], reg.tvalues["RMW"], reg.tvalues["CMA"]
        ]
    # 样本量不足时返回全NaN
    return [np.nan] * 12  # 修正：原函数返回10个NaN，实际需匹配12个输出列

当前问题代码 #

这段代码直接对整个数据集执行滚动窗口，未按Fund分组，导致不同基金的数据被混入同一回归窗口：

df_1 = funds.join(
    pd.DataFrame(
        (process(x) for x in funds.rolling(t_window)),
        columns=["alpha", "Beta_RMRF", "Beta_SMB", "Beta_HML", "Beta_RMW", "Beta_CMA",
                 "t_alpha", "t_RMRF", "t_SMB", "t_HML", "t_RMW", "t_CMA"]
    )
)

修正后的代码 #

核心是通过groupby('Fund')对每个基金单独处理，再应用滚动窗口和自定义函数：

# 第一步：确保数据按基金和时间排序（假设存在时间列Date，需根据实际数据调整列名）
funds = funds.sort_values(['Fund', 'Date']).reset_index(drop=True)

# 第二步：按基金分组执行滚动窗口回归
# raw=False 确保传入process的是DataFrame而非numpy数组
grouped_results = funds.groupby('Fund').rolling(t_window).apply(process, raw=False)

# 第三步：将分组结果展开并与原数据合并
result_cols = ["alpha", "Beta_RMRF", "Beta_SMB", "Beta_HML", "Beta_RMW", "Beta_CMA",
               "t_alpha", "t_RMRF", "t_SMB", "t_HML", "t_RMW", "t_CMA"]
df_1 = funds.join(
    pd.DataFrame(grouped_results.tolist(), columns=result_cols, index=funds.index),
    how='left'
)

注意事项 #

1. 必须确保数据按基金和时间排序，否则滚动窗口逻辑会混乱
2. raw=False参数不可省略，否则process函数无法正确处理窗口数据
3. 示例数据中每个基金仅4行，远小于窗口大小36，因此运行示例会全返回NaN，实际数据集满足窗口条件后会输出有效结果
4. 针对10万+行的大数据量，若运行速度较慢，可考虑引入并行分组处理（如swifter库）优化效率

内容的提问来源于stack exchange，提问作者MF1992