使用Jsoup/scalaj.http请求接口遇500错误，求排查与解决

使用的版本信息 #

scala version: "2.13.1"
sbt version: "1.2.8"
jsoup version: "1.15.3"

初始尝试的Scala代码 #

目标接口https://firststop.sos.nd.gov/api/Records/notarysearch要求POST JSON格式请求体：{"SEARCH_VALUE":"ab","STARTS_WITH_YN":false,"ACTIVE_ONLY_YN":false,"ELECTRONIC_NOTARY_ONLY_YN":false,"REMOTE_NOTARY_ONLY_YN":false}，但初始代码调用后返回500错误：

val homePageUrl = "https://firststop.sos.nd.gov/search/notary"
val searchPage = "https://firststop.sos.nd.gov/api/Records/notarysearch"
val jsoup = Jsoup.connect(searchPage)

val response = jsoup.data("ACTIVE_ONLY_YN","false" )
  .data("SEARCH_VALUE", "ab")
  .data("ELECTRONIC_NOTARY_ONLY_YN", "0")
  .data("REMOTE_NOTARY_ONLY_YN", "false")
  .data("STARTS_WITH_YN", "false")
  .post()
println(response)

初始错误信息 #

Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=500, URL=[https://firststop.sos.nd.gov/api/Records/notarysearch]
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:890)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:829)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:366)
    at org.jsoup.helper.HttpConnection.post(HttpConnection.java:360)

已尝试的调整方案 #

调整1：设置User-Agent和超时参数 #

添加浏览器User-Agent、设置超时、忽略错误与内容类型，仍返回500错误：

val jsoup = Jsoup.connect(searchPageUrl).userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36").timeout(0).ignoreHttpErrors(true).ignoreContentType(true).followRedirects(true)

val response = jsoup.data("ACTIVE_ONLY_YN","false" )
  .data("SEARCH_VALUE", "ab")
  .data("ELECTRONIC_NOTARY_ONLY_YN", "0")
  .data("REMOTE_NOTARY_ONLY_YN", "false")
  .data("STARTS_WITH_YN", "false")
  .post()

println(response)

得到的错误响应：

<html>
 <head></head>
 <body>
  {"code":"api/Records","message":"Internal Error Occurred","internalerror":null,"title":""}
 </body>
</html>

调整2：直接发送JSON请求体 #

设置Content-Type: application/json并传递JSON格式请求体，结果还是500错误：

val payload = """{SEARCH_VALUE:"ab",STARTS_WITH_YN:false,ACTIVE_ONLY_YN:false,ELECTRONIC_NOTARY_ONLY_YN:false,REMOTE_NOTARY_ONLY_YN:false}""".stripMargin

val request: Connection.Response = Jsoup.connect(searchPageUrl)
  .header("Content-Type", "application/json")
  .method(Connection.Method.POST)
  .ignoreContentType(true)
  .requestBody(payload)
  .execute()
val responseBody = request.body()
println(responseBody)

调整3：改用scalaj.http库 #

换用scalaj.http发送请求，依旧返回500错误：

val payload = """{SEARCH_VALUE:"ab",STARTS_WITH_YN:false,ACTIVE_ONLY_YN:false,ELECTRONIC_NOTARY_ONLY_YN:false}""".stripMargin

val response = Http(searchPageUrl).postData(payload).header("Content-Type", "application/json").asString
println(response)

val responseBody = response.body
val json = responseBody.parseJson
println(json)

得到的错误响应：

HttpResponse({"code":"api/Records","message":"Internal Error Occurred","internalerror":null,"title":""},500,TreeMap(Access-Control-Allow-Headers -> Vector(Origin, Content-Type, Accept, Content-Encoding, Authorization), Access-Control-Allow-Methods -> Vector(*), Access-Control-Allow-Origin -> Vector(*), Access-Control-Expose-Headers -> Vector(session-timeout, Request-Context), Cache-Control -> Vector(no-cache), Connection -> Vector(close), Content-Length -> Vector(90), Content-Type -> Vector(application/json; charset=utf-8), Date -> Vector(Wed, 30 Aug 2023 07:29:59 GMT), Expires -> Vector(-1), Pragma -> Vector(no-cache), Request-Context -> Vector(appId=cid-v1:df24017c-37e9-4e1c-afab-260d80eaaeea), Server -> Vector(State of North Dakota), session-timeout -> Vector(0), Set-Cookie -> Vector(ASP.NET_SessionId=j3ioe0kyrrkaqrwlymi0dq0r; path=/; HttpOnly; SameSite=Lax), Status -> Vector(HTTP/1.1 500 Internal Server Error), X-AspNet-Version -> Vector(4.0.30319), X-Content-Type-Options -> Vector(nosniff), X-XSS-Protection -> Vector(1;  mode=block)))
{"code":"api/Records","internalerror":null,"message":"Internal Error Occurred","title":""}

疑问 #

• 我的操作哪里存在错误？
• 还有其他获取该网站数据的可行方法吗？

内容的提问来源于stack exchange，提问作者user21236160