如何将函数批量应用于多个API响应列表以提取并合并美国游说数据

Hey there! Let's sort out this data extraction issue for you. The main problem with your current code is that using assign() to store each API response as a separate URL-named object makes it really tough to process them as a group. Instead, we'll refactor the workflow to keep everything organized in lists, which plays perfectly with lapply().

Step 1: Cleanly Generate Your API URLs #

First, let's fix how you create the list of URLs—no need for sample() here, since we want a sequential list of pages:

# Define your API parameters clearly
base_url <- "https://lda.senate.gov/api/v1/contributions/?page="
total_pages <- 10  # Adjust this to your total needed pages
page_size <- 250

# Generate all page URLs in one vector
urls <- paste0(base_url, 1:total_pages, "&page_size=", page_size)

Step 2: Fetch All Responses into a List #

Instead of scattering responses across separate objects, we'll store them all in a single list using lapply():

# Load the httr package (make sure it's installed first: install.packages("httr"))
library(httr)

# Fetch every URL and store responses in a list
responses <- lapply(urls, GET)

Step 3: Extract Data Frames from Each Response #

Now we can use lapply() again to parse each JSON response into a data frame. Note: Most APIs return data under a key like results—you might need to verify this by checking one response's structure first (e.g., fromJSON(rawToChar(responses[[1]]$content)) to inspect the JSON):

# Load jsonlite (install with: install.packages("jsonlite"))
library(jsonlite)

# Parse each response into a data frame
data_frames <- lapply(responses, function(resp) {
  # Convert raw content to JSON
  parsed_json <- fromJSON(rawToChar(resp$content))
  # Extract the actual data table (adjust "results" to match your API's key if needed)
  parsed_json$results
})

Step 4: Combine All Data Frames #

Now that you have a list of data frames, merging them is straightforward:

# Option 1: Base R approach
full_dataset <- do.call(rbind, data_frames)

# Option 2: Using dplyr for more flexibility (install.packages("dplyr"))
# library(dplyr)
# full_dataset <- bind_rows(data_frames)

Optional: Verify Request Success #

It's a good idea to check that all API requests went through properly:

# Get status codes for all responses
status_codes <- sapply(responses, function(resp) resp$status_code)

if (all(status_codes == 200)) {
  message("All API requests succeeded!")
} else {
  warning(paste("Uh-oh,", sum(status_codes != 200), "requests failed—check those URLs!"))
}

Quick note on why your original lapply() didn't work: You were trying to apply the function to scattered, individually named objects instead of a single list of responses. By keeping all responses grouped in a list, lapply() can easily iterate over each one without hassle.

内容的提问来源于stack exchange，提问作者Michael Taffe