未用Docker等待JSON API加载，R+jsonlite爬取政府数据不全问题

安全阿岳

2026-5-26

Fixing Pagination Limits for Your Medicare Data Scrape

Hey there! The issue you're hitting is super common with open government APIs—they almost always enforce pagination limits (here, a 1000-row cap per request) to prevent their servers from being flooded. Luckily, this is easy to work around entirely locally (no remote servers needed) by leveraging the API's built-in pagination parameters.

Why You're Only Getting 1000 Rows

The Medicare data API uses the Socrata platform, which defaults to returning just 1000 records per request. To get all ~32,000 rows, you need to explicitly request pages using the $limit (rows per page) and $offset (starting position) parameters.

Step-by-Step Solution Code

Here's how to modify your code to fetch every record:

library(jsonlite)

# First, grab the total number of records to know how many pages we need
total_records_url <- "https://data.medicare.gov/resource/rmgi-5fhi.json?$select=count(*)"
total_records <- fromJSON(total_records_url)$count

# Configure pagination settings (keep 1000 rows per page to match the API's max)
rows_per_page <- 1000
page_offsets <- seq(from = 0, to = total_records - 1, by = rows_per_page)

# Helper function to fetch a single page of data
fetch_single_page <- function(offset) {
  page_url <- paste0(
    "https://data.medicare.gov/resource/rmgi-5fhi.json?",
    "$limit=", rows_per_page,
    "&$offset=", offset
  )
  fromJSON(page_url)
}

# Fetch all pages and combine into one data frame
full_hcahps_data <- do.call(rbind, lapply(page_offsets, fetch_single_page))

# Verify you have all the data
dim(full_hcahps_data)

Extra Tips

If you run into rate-limiting issues (unlikely for non-commercial use with this API), add a small delay between requests by modifying the helper function:

fetch_single_page <- function(offset) {
  page_url <- paste0(
    "https://data.medicare.gov/resource/rmgi-5fhi.json?",
    "$limit=", rows_per_page,
    "&$offset=", offset
  )
  data <- fromJSON(page_url)
  Sys.sleep(1) # Wait 1 second between requests
  data
}