Apache Spark消费RESTful API转DataFrame遇json4s映射异常求助

Hey there, let's break down what's causing that error and fix it right away.

First, looking at your error message:

Caused by: org.json4s.package$MappingException: Expected collection but got JObject(...)

This is a classic case of mismatched JSON structure vs. your parsing code. The Bittrex API returns a single JSON object (not an array), but your code is trying to extract it as an Array[Result]—that's why json4s is yelling that it expected a collection but got a single object.

Let's confirm the API's response structure: it sends back one top-level object with three fields: success, message, and result (where result is the array of market data you care about). Your Result case class is actually correct for this structure, but the parsing step was wrong.

Here's the fixed code with explanations:

package trials

import org.apache.spark.sql.SparkSession
import org.json4s.jackson.JsonMethods.parse
import scala.io.Source.fromURL

object DEF {
  implicit val formats = org.json4s.DefaultFormats
  
  case class Result(success: Boolean, message: String, result: Array[Markets])
  case class Markets(
    MarketCurrency: String,
    BaseCurrency: String,
    MarketCurrencyLong: String,
    BaseCurrencyLong: String,
    MinTradeSize: Double,
    MarketName: String,
    IsActive: Boolean,
    Created: String,
    Notice: String,
    IsSponsored: Option[String], // Updated to handle null values from the API
    LogoUrl: Option[String]      // Same here—nulls will map to None instead of breaking parsing
  )

  def main(args: Array[String]): Unit = {
    val spark = SparkSession
      .builder()
      .appName(s"${this.getClass.getSimpleName}")
      .config("spark.sql.shuffle.partitions", "4")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

    // Parse the top-level Result object (not an array!)
    val parsedData = parse(fromURL("https://bittrex.com/api/v1.1/public/getmarkets").mkString).extract[Result]
    
    // Pull out the actual markets array from the Result and convert to Dataset
    val mySourceDataset = spark.createDataset(parsedData.result)
    
    mySourceDataset.printSchema()
    mySourceDataset.show()
  }
}

Key changes made:

• Fixed the parsing target: Changed extract[Array[Result]] to extract[Result] since the API returns one top-level object, not an array of them.
• Handled null values: Updated IsSponsored and LogoUrl to Option[String]—the API sometimes returns null for these fields, and using Option lets json4s map those nulls to None instead of throwing another mapping error.
• Minor cleanup: Added parentheses to printSchema() for proper Scala method call syntax.

To recap the root cause: Your code expected an array of Result objects, but the API sends a single Result object containing the array of markets. Mismatching that structure is exactly what triggered the "Expected collection but got JObject" error.

内容的提问来源于stack exchange，提问作者Arshanvit