如何编写函数获取需交互操作的网站数据集？（R+SAS PROC IML）

Hey there! I’ve been in your shoes—scraping non-API sites can feel like hitting a wall, especially when your go-to automation tool is no longer available. Let’s walk through two solid solutions that work for your setup (running R via SAS PROC IML):

1. Simulate a Form POST Request (No Browser Automation Needed) #

Even if you didn’t catch the request in Chrome’s Network tab, the submit button is almost certainly sending a POST request under the hood—this site looks like it’s built with ASP.NET, which often uses hidden form fields that can make requests easy to miss. Here’s how to replicate the form submission programmatically:

• First, use rvest and httr to inspect the page’s form structure:

  library(rvest)
  library(httr)
  
  # Fetch the target page
  base_url <- "https://www.treasurydirect.gov/GA-FI/FedInvest/selectSecurityPriceDate.htm"
  page_session <- html_session(base_url)
  form <- html_form(page_session)[[1]] # Grab the main form on the page
  
  # Print the form to see required fields (look for date inputs + hidden ASP.NET fields)
  print(form)
  

• You’ll notice hidden fields like __VIEWSTATE, __VIEWSTATEGENERATOR, and __EVENTVALIDATION—these are critical for the site to accept your request. Don’t omit them!
• Fill in your date values and submit the form:

  # Replace with your actual date list (adjust field names to match the form)
  filled_form <- set_values(form,
                            "priceDate" = "2024-05-20", # Example date format
                            # Add other date fields if needed for your list
                           )
  
  # Submit the form and get the response
  response <- submit_form(page_session, filled_form)
  
  # Extract the results table from the response page
  results_table <- response %>% 
    html_element("table") %>% # Target the results table
    html_table(header = TRUE) # Convert to a data frame
  

• If submit_form() fails, you can manually build the POST request using httr::POST(), passing all form fields (including hidden ones) in the body argument.

2. Use Playwright for R as a Replacement for RSelenium #

Since RSelenium is deprecated, Playwright for R is a modern, well-maintained alternative that handles JavaScript-heavy sites seamlessly. It’s easier to set up and supports all major browsers.

Here’s how to implement it:

• First, install the package and required browser binaries:

  install.packages("playwright")
  library(playwright)
  pw_install() # Installs Chrome/Firefox/Edge binaries
  

• Write automation code to navigate, input dates, submit, and extract data:

  # Launch a headless Chrome session (set headless=FALSE to see the browser)
  pw <- playwright$launch()
  browser <- pw$chromium$launch(headless = TRUE)
  page <- browser$new_page()
  
  # Navigate to the page
  page$goto("https://www.treasurydirect.gov/GA-FI/FedInvest/selectSecurityPriceDate.htm")
  
  # Input your dates (use Chrome DevTools to find the correct selectors for inputs)
  page$fill("#priceDate", "2024-05-20") # Replace selector with your target input
  
  # Click the submit button (adjust selector to match the site's button)
  page$click('input[name="submitButton"]')
  
  # Wait for the results table to load
  page$wait_for_selector("table")
  
  # Extract the table data and convert to a data frame
  table_data <- page$eval_handle("document.querySelector('table')")$evaluate_node(function(node) {
    const rows = Array.from(node.querySelectorAll('tr'));
    return rows.map(row => Array.from(row.querySelectorAll('td, th')).map(cell => cell.textContent.trim()));
  }) %>% as.data.frame()
  
  # Clean up the session
  browser$close()
  pw$stop()
  

Quick Tips for SAS PROC IML #

• Make sure all required R packages (rvest, httr, playwright) are installed in the R instance that SAS uses. Verify this with:

  proc iml;
    submit / r;
      installed.packages();
    endsubmit;
  run;
  

• For large date lists, batch your requests to avoid overwhelming the site—always respect the site’s robots.txt and terms of service.

内容的提问来源于stack exchange，提问作者liberalartsthot