Spark CatalystSqlParser无法导入使用，求同类替代解析工具

Problem Context #

You originally used CatalystSqlParser to convert input strings to Spark's DataType with this code:

private def convertToDataType(inputType: String): DataType = CatalystSqlParser.parseDataType(inputType)

But now the import org.apache.spark.sql.catalyst.parser.CatalystSqlParser is failing, and you need a functional replacement.

Recommended Alternatives #

1. Use DataTypeParser (Spark 3.x+) #

Spark provides a dedicated DataTypeParser class in the public API that acts as a great drop-in replacement for most use cases. It directly parses SQL-style type strings into DataType instances.

First, import the parser:

import org.apache.spark.sql.types.DataTypeParser

Then update your conversion method:

private def convertToDataType(inputType: String): DataType = DataTypeParser.parse(inputType)

This works seamlessly for standard SQL types like int, string, struct<id:int,name:string>, array<double>, and more.

2. Parse via SparkSession SQL Query #

If you have access to a SparkSession instance, you can leverage Spark's core SQL engine to parse the type indirectly. This is especially handy for complex or custom types that might need full SQL engine support.

Here's how to implement it:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types.DataType

private def convertToDataType(inputType: String)(implicit spark: SparkSession): DataType = {
  spark.sql(s"SELECT CAST(NULL AS $inputType)")
    .schema
    .fields(0)
    .dataType
}

Just ensure an implicit SparkSession is available in scope when calling this method.

3. Use DataType.fromJson (For JSON-formatted Type Strings) #

If you can transform your original SQL type string into Spark's JSON schema format, you can use the fromJson static method on DataType. For example, the SQL type struct<id:int,name:string> translates to the JSON string {"type":"struct","fields":[{"name":"id","type":"integer"},{"name":"name","type":"string"}]}.

Example usage:

import org.apache.spark.sql.types.DataType

private def convertToDataType(jsonTypeStr: String): DataType = DataType.fromJson(jsonTypeStr)

Note this requires your input to follow Spark's JSON schema specification, so it's best suited if you can easily convert your original SQL type string to this format.

内容的提问来源于stack exchange，提问作者samba