Snowpark Scala迁移Spark项目时UDF报错：未知用户定义函数

背景 #

刚接触Snowpark，正在将Spark-Scala项目迁移至Snowpark Scala。定义了掩码列的UDF后调用出现SQL编译错误，提示未知UDF；尝试编写简单UDF测试，仍出现同类错误。

掩码列UDF定义 #

def maskColumnsUDF(maskChar : String = "*", noOfLeftVisibleChars : Int = 1, noOfRightVisiblechars : Int = 1): UserDefinedFunction = udf((attrVal :String) => {

      var finalString =  ""
      if(attrVal == null){ attrVal }
      else if (attrVal.length > 4 ) {
        val attrArray = attrVal.toCharArray
        for (counter <- 0 to attrArray.length() - 1) {
          if (counter < noOfLeftVisibleChars || counter > attrArray.length() - (noOfRightVisiblechars + 1)) {
            finalString = finalString + attrArray(counter)
          }
          else {
            finalString = finalString + maskChar
          }
        }
        finalString
      }
      else { "***" }
})

UDF调用代码 #

columnNamesList.foreach(columnName => {
    sampleDf = sampleDf.withColumn(columnName+"_zs_masked",
    UserDefinedFunctions.maskColumnsUDF("*",1,1)(col(columnName)))
})

报错信息 #

[main] ERROR com.snowflake.snowpark.internal.ServerConnection - Failed to analyze schema of query:
Exception in thread "main" net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error: Unknown user-defined function SNOWPARK_TEST.PUBLIC.SNOWPARK_TEMP_FUNCTION_GICGX7MLYN3OHZO

简化测试案例 #

测试DataFrame定义 #

val schema = StructType(Seq(
    StructField("column_name", StringType, nullable = false),
    StructField("featureName", StringType, nullable = false),
    StructField("distribution", DoubleType, nullable = false)
  ))

val data = Seq(
    Row("A", "foo", 0.2),
    Row("A", "bar", 0.5),
    Row("B", "baz", 0.1),
    Row("B", "qux", 0.3)
)

var featureDF: DataFrame = session.createDataFrame(data, schema)

测试UDF调用 #

featureDF = featureDF.withColumn("test_udf", UserDefinedFunctions.test_udf(col("column_name")))

简化测试报错信息 #

[main] ERROR com.snowflake.snowpark.internal.ServerConnection - failed to execute query:
Exception in thread "main" net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error:
Unknown user-defined function SNOWPARK_TEST.PUBLIC.SNOWPARK_TEMP_FUNCTION_R0KRMUFIRRFYNUX

内容的提问来源于stack exchange，提问作者Hamdan Zaheer