sparklyr中ft_normalizer与ft_min_max_scaler报错问题求助

Hey, great catch on suspecting data type issues—you’re exactly right! The problem here is that Spark ML's ft_normalizer and ft_min_max_scaler expect input columns to be vector types (specifically VectorUDT), but your dep_delay column is a plain numeric DoubleType. Unlike ft_binarizer which supports direct numeric input, these scaling functions are designed to handle multi-feature vectors (even if you're only working with a single feature).

解决方案步骤 #

1. 将单个数值列转换为向量列：使用ft_vector_assembler把你的dep_delay列打包成一个单元素向量列。
2. 运行缩放器函数：用转换后的向量列作为输入，即可正常调用ft_normalizer和ft_min_max_scaler。
3. 可选：将向量结果转回数值列：如果后续需要普通数值格式，可以提取向量中的元素。

修改后的完整代码 #

library(sparklyr)
library(dplyr)
library(nycflights13)

# 连接本地Spark集群
sc <- spark_connect(master = "local", version = "2.1.0")

# 准备数据并上传到Spark
x <- flights %>% select(dep_delay)
x_tbl <- sdf_copy_to(sc, x)

# 第一步：把Double类型的列转换为Vector类型
x_vector_tbl <- ft_vector_assembler(
  x = x_tbl,
  input.cols = "dep_delay",
  output.col = "dep_delay_vec"
)

# 现在ft_normalizer可以正常运行了
normalized_tbl <- ft_normalizer(
  x = x_vector_tbl,
  input.col = "dep_delay_vec",
  output.col = "delayed_norm"
)

# ft_min_max_scaler同样可以正常运行
min_max_scaled_tbl <- ft_min_max_scaler(
  x = x_vector_tbl,
  input.col = "dep_delay_vec",
  output.col = "delayed_min_max"
)

# 查看结果（向量列会显示为[数值]格式）
normalized_tbl %>% 
  select(dep_delay, delayed_norm) %>% 
  head()

min_max_scaled_tbl %>% 
  select(dep_delay, delayed_min_max) %>% 
  head()

# 可选：将向量列转回普通数值列
normalized_tbl <- normalized_tbl %>%
  mutate(delayed_norm_val = invoke(delayed_norm, "apply", 0))

# 断开Spark连接
spark_disconnect(sc)

错误原因补充 #

从你的报错信息里也能验证这点：

• ft_min_max_scaler直接提示：Column dep_delay must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually DoubleType
• ft_normalizer的错误是因为它试图把Double值转换成向量，导致用户定义函数执行失败。

内容的提问来源于stack exchange，提问作者Flo585