使用Dask处理超内存数据集的Lasso回归实现难题

我需要在超内存数据集上运行Lasso回归，尝试了两种dask-glm的方案，但都存在系数不稳定、结果与scikit-learn不一致的问题，且未找到原生Dask的完善实现方案（仅找到将Dask作为joblib后端的LassoCV方案，不符合需求）。

尝试的方案及问题 #

方案1：dask-glm梯度下降+ElasticNet正则 #

from dask_ml.datasets import make_regression
import dask_glm.families
import dask_glm.regularizers
import dask_glm.algorithms
import pandas as pd
from sklearn import linear_model

# Dask数据集
X, y = make_regression(n_samples=1000, chunks=100)

# Pandas数据集用于对比
df_X = X.compute()
df_y = y.compute()

family = dask_glm.families.Normal()
regularizer = dask_glm.regularizers.ElasticNet(weight=1)
b = dask_glm.algorithms.gradient_descent(X=X, y=y, max_iter=100000, family=family, regularizer=regularizer, alpha=0.01, normalize=False, fit_intercept=False)
print(b)

reg = linear_model.Lasso(alpha=0.01, fit_intercept=False)
reg.fit(df_X, df_y)
print(reg.coef_)

问题：Dask计算的系数与Sklearn完全不符，且多次运行结果波动极大。

方案2：dask-glm LinearRegression+L1正则 #

from dask_ml.datasets import make_regression
from dask_glm.regularizers import L1
from dask_glm.estimators import LinearRegression

X, y = make_regression(n_samples=1000, chunks=100)

lr = LinearRegression(regularizer=L1())
lr.fit(X, y)
print(lr.coef_)

问题：系数依然极不稳定，结果不可靠。

1. 调整dask-glm梯度下降参数，优化收敛性 #

dask-glm的梯度下降默认参数对L1正则的支持较差，通过调整以下参数可提升稳定性和一致性：

• 对齐归一化设置：Sklearn的Lasso默认normalize=True，需同步设置Dask代码的normalize=True
• 降低学习率：默认学习率过高会导致震荡，设置learning_rate=0.001或使用自适应学习率learning_rate='adaptive'
• 收紧收敛阈值：设置tol=1e-7，确保算法充分收敛
• 固定随机种子：添加random_state保证结果可复现

调整后的代码：

from dask_ml.datasets import make_regression
import dask_glm.families
import dask_glm.regularizers
import dask_glm.algorithms
from sklearn import linear_model

X, y = make_regression(n_samples=1000, chunks=100, random_state=42)
df_X = X.compute()
df_y = y.compute()

family = dask_glm.families.Normal()
regularizer = dask_glm.regularizers.ElasticNet(weight=1)
b = dask_glm.algorithms.gradient_descent(
    X=X, y=y, 
    max_iter=10000, 
    family=family, 
    regularizer=regularizer, 
    alpha=0.01, 
    normalize=True,
    fit_intercept=False,
    learning_rate=0.001,
    tol=1e-7,
    random_state=42
)
print("Dask梯度下降系数:", b)

reg = linear_model.Lasso(alpha=0.01, fit_intercept=False, random_state=42)
reg.fit(df_X, df_y)
print("Sklearn系数:", reg.coef_)

2. 使用dask-ml原生Lasso实现（推荐） #

dask-ml提供了分布式的Lasso实现，基于坐标下降法，天生适配L1正则，稳定性和与Sklearn的一致性远优于dask-glm的梯度下降方案，且原生支持超内存数据集：

from dask_ml.datasets import make_regression
from dask_ml.linear_model import Lasso
from sklearn.linear_model import Lasso as SklearnLasso

X, y = make_regression(n_samples=1000, chunks=100, random_state=42)

# Dask分布式Lasso
lasso = Lasso(alpha=0.01, fit_intercept=False, max_iter=1000, tol=1e-7, random_state=42)
lasso.fit(X, y)
print("Dask-ML Lasso系数:", lasso.coef_)

# Sklearn对比
df_X = X.compute()
df_y = y.compute()
sk_lasso = SklearnLasso(alpha=0.01, fit_intercept=False, random_state=42)
sk_lasso.fit(df_X, df_y)
print("Sklearn Lasso系数:", sk_lasso.coef_)

关键注意点 #

• dask-glm的梯度下降法更适合L2正则（Ridge）场景，L1正则优先选择dask-ml的实现
• 处理超内存数据集时，dask-ml会自动分块并行计算，无需手动加载全量数据到内存
• 始终设置random_state，便于调试和复现结果

内容的提问来源于stack exchange，提问作者valkmit