ASPX页面爬取进阶求助：获取科罗拉多州执照详情及资料

我需要从科罗拉多州执照查询页面爬取数据，流程为：输入执照ID（例如66244），点击“Detail”进入详情页，再点击“view profile”进入个人资料页。目前我用httpx编写了代码，能获取到包含Detail按钮的响应，但不知道如何处理该按钮对应的JavaScript链接，无法进入详情页并获取个人资料页的链接，请求技术帮助。

我的代码 #

import httpx
from selectolax.parser import HTMLParser


def getLicense(response, number):

  html = HTMLParser(response.text) 
  viewstate = html.css_first("input#__VIEWSTATE").attributes['value']
  generator = html.css_first("input#__VIEWSTATEGENERATOR").attributes['value']

  formdata = {
      "ctl00$ScriptManager1":
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$UpdtPanelGridLookup",
      "__EVENTTARGET":
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$UpdtPanelGridLookup",
      "__EVENTARGUMENT":
      1,
      "__VIEWSTATE":
      viewstate,
      "__VIEWSTATEGENERATOR":
      generator,
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$ddCredPrefix":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbLicenseNumber":
      "66244",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$ddSubCategory":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbFirstName_Contact":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbLastName_Contact":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbDBA_Contact":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbMaidenName_Contact":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbCity_ContactAddress":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$ddStates":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ctl03$tbZipCode_ContactAddress":
      "",
      "ctl00$MainContentPlaceHolder$ucLicenseLookup$ResizeLicDetailPopupID_ClientState":
      "0,0",
      "ctl00$OutsidePlaceHolder$ucLicenseDetailPopup$ResizeLicDetailPopupID_ClientState":
      "0,0",
      "__ASYNCPOST":
      "false"
  }

  yield formdata


def main():

  
  header = { "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36", "Cookie": "ASP.NET_SessionId=ycpyzckej2z3pq1pltjpt55k; apps.colorado.gov=!qseyvwywknmjz6ywzsK/zCjMlNMem+DnBFEVDa2UErNVnVk/IwmqktUUbE9Sk9Ed2SwBEeqRdai5GHM=" }
  
  client = httpx.Client() 
  url = "https://apps2.colorado.gov/dora/licensing/lookup/" 
  data = client.get(url + "licenselookup.aspx")  
  
  for license in getLicense(data, 66244): 
    
    licenseData = client.post(url + "licenselookup.aspx", data=license, headers = header)
    
    
    with open('license_data.html', 'w') as file:
      file.seek(0)
      file.write(licenseData.text)


if __name__ == "__main__":
  main()

响应中的Detail按钮代码 #

<a id="ctl00_MainContentPlaceHolder_ucLicenseLookup_gvSearchResults_ctl03_HyperLinkDetail" class="btn btn-primary btn-xs" href="javascript:DisplayLicenceDetail(&#39;1361971;1460049;0;Michael John Abadier;133644907;0&#39;)">Detail</a>

步骤1：解析JavaScript参数 #

先从DisplayLicenceDetail函数的参数中提取关键数据。参数是分号分隔的字符串，拆分后即可拿到所需值：

# 在获取licenseData后添加这段代码
detail_html = HTMLParser(licenseData.text)
detail_btn = detail_html.css_first("#ctl00_MainContentPlaceHolder_ucLicenseLookup_gvSearchResults_ctl03_HyperLinkDetail")
if detail_btn:
    js_href = detail_btn.attributes['href']
    # 提取括号内的参数字符串，去除转义单引号
    param_str = js_href.split("('")[1].split("')")[0].replace("'", "'")
    params = param_str.split(';')
    # params数组包含了详情页所需的核心识别参数

步骤2：构造详情页POST请求 #

该网站的Detail按钮点击后通过AJAX异步加载内容，需要构造符合ASPX规范的POST表单，核心是维护最新的__VIEWSTATE、__VIEWSTATEGENERATOR，以及指定正确的事件目标：

# 获取页面最新的VIEWSTATE参数
viewstate = detail_html.css_first("input#__VIEWSTATE").attributes['value']
generator = detail_html.css_first("input#__VIEWSTATEGENERATOR").attributes['value']

# 构造详情请求的表单数据
detail_form = {
    "__EVENTTARGET": "ctl00$MainContentPlaceHolder$ucLicenseLookup$gvSearchResults",
    "__EVENTARGUMENT": "Select$2",  # ctl03对应行索引2（GridView从0开始计数）
    "__VIEWSTATE": viewstate,
    "__VIEWSTATEGENERATOR": generator,
    "ctl00$ScriptManager1": "ctl00$MainContentPlaceHolder$ucLicenseLookup$UpdtPanelGridLookup|ctl00$MainContentPlaceHolder$ucLicenseLookup$gvSearchResults",
    "__ASYNCPOST": "true",
    "ctl00$MainContentPlaceHolder$ucLicenseLookup$ResizeLicDetailPopupID_ClientState": "0,0",
    "ctl00$OutsidePlaceHolder$ucLicenseDetailPopup$ResizeLicDetailPopupID_ClientState": "0,0"
}

# 发送请求获取详情内容
detail_response = client.post(url + "licenselookup.aspx", data=detail_form, headers=header)

步骤3：提取个人资料页链接 #

解析详情页响应，找到View Profile的跳转链接，直接请求即可获取个人资料：

profile_html = HTMLParser(detail_response.text)
profile_link = profile_html.css_first("a[href*='ViewProfile']")
if profile_link:
    profile_url = url + profile_link.attributes['href']
    # 请求个人资料页
    profile_response = client.get(profile_url, headers=header)
    # 此处可添加对profile_response内容的解析逻辑

关键注意事项 #

• 全程使用同一个httpx.Client实例，自动维护会话Cookie和状态，避免请求失效。
• 每次POST请求后必须重新获取最新的__VIEWSTATE和__VIEWSTATEGENERATOR，ASPX页面会实时更新这些值。
• __EVENTARGUMENT中的索引需与按钮所在行的序号对应（从0开始计数）。

内容的提问来源于stack exchange，提问作者Soren Hubble