Spark 3.3.0中ALS类的setMaxIter等方法找不到问题求助

问题背景 #

使用Spark 3.3.0 + Scala 2.13编写基于协同过滤的推荐系统时，原代码中直接实例化ALS类后调用setMaxIter、setRegParam和fit方法会报错，提示这些方法不存在。

原因 #

Spark 3.x版本对ALS的API进行了重构，参数配置和模型训练的入口改为ALS.Builder模式，原直接通过ALS实例调用的方法已被移除。

修正后的完整代码 #

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.Encoders;
import org.apache.spark.ml.recommendation.ALS;
import org.apache.spark.ml.recommendation.ALSModel;
import java.util.Collections;

public class CollaborativeFiltering {
    public static void main(String[] args) {
        // Step 1: Set up Spark environment
        SparkSession spark = SparkSession.builder()
                .appName("CollaborativeFiltering")
                .master("local[*]")
                .getOrCreate();

        // Step 2: Configure database connection and load ratings data into DataFrame
        String url = "jdbc:mysql://localhost:3306/your_database"; // Replace with your database URL
        String table = "ratings"; // Replace with your table name
        String user = "your_username"; // Replace with your database username
        String password = "your_password"; // Replace with your database password
        
        Dataset<Row> ratingsDF = spark.read()
                .format("jdbc")
                .option("url", url)
                .option("dbtable", table)
                .option("user", user)
                .option("password", password)
                .load();

        // Step 3: Prepare data for collaborative filtering
        Dataset<Row> preparedData = ratingsDF.withColumnRenamed("user_id", "userId")
                .withColumnRenamed("product_id", "itemId");

        // Step 4: Build collaborative filtering model using ALS.Builder
        ALS als = ALS.newBuilder()
                .setUserCol("userId")
                .setItemCol("itemId")
                .setRatingCol("rating")
                .setRank(10) // Set the number of latent factors
                .setMaxIter(10) // Set the maximum number of iterations
                .setRegParam(0.01) // Set the regularization parameter
                .build();

        ALSModel model = als.fit(preparedData);

        // Step 5: Generate recommendations for a specific user
        int userId = 123; // Replace with the desired user ID
        Dataset<Row> userRecommendations = model.recommendForUserSubset(
                spark.createDataset(Collections.singletonList(userId), Encoders.INT), 
                5); // Get top 5 recommendations

        // Print the recommendations
        userRecommendations.show(false);
        
        // Stop the Spark session
        spark.stop();
    }
}

关键修改点 #

• 替换new ALS()为ALS.newBuilder()，通过Builder链式设置所有参数
• 调用build()方法生成可用于训练的ALS实例
• 确保导入的ALS类来自org.apache.spark.ml.recommendation包（而非旧的mllib包）

内容的提问来源于stack exchange，提问作者medi