POST请求爬取NYISO夏季数据正常但冬季失败的技术求助

爬取纽约电网（NYISO）公开表格数据，目标网址：http://icap.nyiso.com/ucap/public/auc_view_spot_detail.do。当前通过POST请求可成功获取夏季时段数据，但切换为冬季时段请求体时，无法生成目标表格。

操作流程：需选择Season与Month组合后点击Display生成表格，已复制请求头信息及URL编码负载作为POST请求体。

可正常运行的夏季数据爬取R代码 #

# libraries
library(jsonlite) 
library(lubridate)
library(data.table)
library(httr)
library(rvest)

# get session and cookies
initial_url <- "http://icap.nyiso.com/ucap/public/auc_view_spot_detail.do"
initial_response <- GET(initial_url)
cookie_data <- cookies(initial_response)
cookie_string <- paste0(cookie_data$name, "=", cookie_data$value, collapse = "; ")


# Define the POST request headers, including cookies
headers <- c(
  "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
  "Accept-Encoding" = "gzip, deflate",
  "Accept-Language" = "en-US,en;q=0.9",
  "Cache-Control" = "max-age=0",
  "Connection" = "keep-alive",
  "Content-Length" = "85",
  "Content-Type" = "application/x-www-form-urlencoded",
  "Cookie" = cookie_string,
  "Host" = "icap.nyiso.com",
  "Origin" = "null",
  "Upgrade-Insecure-Requests" = "1",
  "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
)

# Define the URL for the POST request
post_url <- "http://icap.nyiso.com/ucap/public/auc_view_spot_detail.do"

# Below is working code for a "Summer" season:
response <- POST(post_url, add_headers(.headers = headers), encode = "form", 
                 body = "seasonId=702793&seasonId=Summer+2024&month=05%2F2024&month=May%2F2024&display=Display")
html_content <- content(response, as = "text")
html <- read_html(html_content)
tables <- html %>% html_nodes("table")
html_table(tables[4]) # print

夏季请求返回的表格数据 #

#[[1]]
## A tibble: 45 × 2
#   X1                           X2            
#   <chr>                        <chr>         
# 1 "05/2024"                    "05/2024"     
# 2 "G-J Locality"               "G-J Locality"
# 3 "Awarded Deficiency (MW)"    "1,888.1"     
# 4 "Awarded Excess (MW)"        "1,694.400"   
# 5 "% Excess Above Requirement" "14.78"       
# 6 "Price ($/kW-M)"             "$4.27"       
# 7 ""                           ""            
# 8 "LI"                         "LI"          
# 9 "Awarded Deficiency (MW)"    "176.9"       
#10 "Awarded Excess (MW)"        "519.200"     
## ℹ 35 more rows
## ℹ Use `print(n = ...)` to see more rows

无法获取数据的冬季请求代码 #

# does not work to generate the data
response <- POST(post_url, add_headers(.headers = headers), encode = "form", 
                 body = "seasonId=702409&seasonId=Winter+2023-2024&month=02%2F2024&month=Feb%2F2024&display=Display")
html_content <- content(response, as = "text")
html <- read_html(html_content)
tables <- html %>% html_nodes("table")
html_table(tables[4]) # there is no such table

异常现象 #

• 请求体存在重复参数，但移除任意一个均会导致请求失败；
• 只要seasonId对应的字符串（如seasonId=Summer+2024）正确，可替换为任意有效数字ID（其他ID可从http://icap.nyiso.com/ucap/rest/seasons/public获取）；
• 未找到对应表格数据的公开REST API。

测试用请求体字符串（仅冬季时段请求失败） #

body_strings <- c("seasonId=700085&seasonId=Winter+2021-2022&month=01%2F2022&month=Jan%2F2022&display=Display", 
"seasonId=700085&seasonId=Winter+2021-2022&month=02%2F2022&month=Feb%2F2022&display=Display", 
"seasonId=700085&seasonId=Winter+2021-2022&month=03%2F2022&month=Mar%2F2022&display=Display", 
"seasonId=700085&seasonId=Winter+2021-2022&month=04%2F2022&month=Apr%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=05%2F2022&month=May%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=06%2F2022&month=Jun%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=07%2F2022&month=Jul%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=08%2F2022&month=Aug%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=09%2F2022&month=Sep%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=10%2F2022&month=Oct%2F2022&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=11%2F2022&month=Nov%2F2022&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=12%2F2022&month=Dec%2F2022&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=01%2F2023&month=Jan%2F2023&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=02%2F2023&month=Feb%2F2023&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=03%2F2023&month=Mar%2F2023&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=04%2F2023&month=Apr%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=05%2F2023&month=May%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=06%2F2023&month=Jun%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=07%2F2023&month=Jul%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=08%2F2023&month=Aug%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=09%2F2023&month=Sep%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=10%2F2023&month=Oct%2F2023&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=11%2F2023&month=Nov%2F2023&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=12%2F2023&month=Dec%2F2023&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=01%2F2024&month=Jan%2F2024&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=02%2F2024&month=Feb%2F2024&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=03%2F2024&month=Mar%2F2024&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=04%2F2024&month=Apr%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=05%2F2024&month=May%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=06%2F2024&month=Jun%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=07%2F2024&month=Jul%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=08%2F2024&month=Aug%2F2024&display=Display"
)

内容的提问来源于stack exchange，提问作者Evan Friedland