PySpark中LogisticRegressionTrainingSummary无fMeasureByThreshold属性问题求助

你遇到的AttributeError是因为fMeasureByThreshold这个属性在新版本的PySpark中已经被移除了，替代方式是通过Precision-Recall曲线数据来计算不同阈值对应的F1值。

你的原始代码 #

from pyspark.ml.classification import LogisticRegression

# Create initial LogisticRegression model
lr = LogisticRegression(labelCol="label", featuresCol="features", maxIter=10)

# set threshold for the probability above which to predict a 1
lr.setThreshold(train_positive_rate)
# lr.setThreshold(0.5) # could use this if knew you had balanced data

# Train model with Training Data
lrModel = lr.fit(train)

# get training summary used for eval metrics and other params
lrTrainingSummary = lrModel.summary

# Find the best model threshold if you would like to use that instead of the empirical positve rate
fMeasure = lrTrainingSummary.fMeasureByThreshold

错误信息 #

AttributeError: 'LogisticRegressionTrainingSummary' object has no attribute 'fMeasureByThreshold'

替代解决方案 #

1. 获取Precision-Recall曲线数据：从训练摘要中提取包含阈值、精确率、召回率的DataFrame

pr = lrTrainingSummary.pr

2. 计算各阈值对应的F1值：利用精确率和召回率的公式计算F1（F-measure）

from pyspark.sql.functions import col

# 计算F1值，公式为 2*(P*R)/(P+R)
f_measure_df = pr.withColumn("f1", 2 * (col("precision") * col("recall")) / (col("precision") + col("recall")))

3. 查看结果：f_measure_df包含了每个阈值对应的精确率、召回率和F1值，你可以根据需求筛选或排序，比如找到F1值最高的阈值：

best_f1_row = f_measure_df.orderBy(col("f1").desc()).first()
best_threshold = best_f1_row.threshold
best_f1 = best_f1_row.f1

额外提示 #

如果不确定当前版本的LogisticRegressionTrainingSummary支持哪些属性，可以用dir(lrTrainingSummary)查看所有可用的方法和属性，避免使用已废弃的API。

内容的提问来源于stack exchange，提问作者Jatin Rathour