基于Tidyverse的API数据抽取：分页数据批量获取技术问询

Hey there! Nice work getting the initial timesheet extraction up and running—pagination is one of those universal API hurdles, and I’ve got a tidyverse-friendly approach that’ll keep your code clean, readable, and easy to maintain.

Here’s a step-by-step solution tailored to your goal of fetching all paginated timesheet data:

1. Core Setup & Dependencies #

First, make sure you’ve got these packages installed (they’re all part of the tidyverse ecosystem or play nicely with it):

• httr for making API requests
• dplyr for data manipulation
• purrr for iterating over pages
• glue for clean string interpolation
• jsonlite for parsing API responses

library(tidyverse)
library(httr)
library(jsonlite)
library(glue)

2. Define a Single-Page Fetch Function #

Start with a reusable function to grab data from a single page. This keeps your code modular—you can tweak the endpoint or parameters without touching the pagination logic later.

fetch_harvest_timesheet_page <- function(page = 1, per_page = 100) {
  # Replace with your Harvest API credentials and account ID
  harvest_account_id <- "YOUR_ACCOUNT_ID"
  harvest_token <- "YOUR_API_TOKEN"
  
  response <- GET(
    url = glue("https://api.harvestapp.com/v2/time_entries"),
    add_headers(
      `Harvest-Account-ID` = harvest_account_id,
      Authorization = glue("Bearer {harvest_token}"),
      `User-Agent` = "Your Company Reporting System"
    ),
    query = list(
      page = page,
      per_page = per_page
    )
  )
  
  # Check for successful response
  stop_for_status(response)
  
  # Parse JSON response into a tibble
  response %>%
    content(as = "text") %>%
    fromJSON() %>%
    pluck("time_entries") %>%
    as_tibble()
}

3. Build the Pagination Logic #

Harvest’s API returns pagination metadata in the response headers (like X-Total-Pages). We’ll use that to figure out how many pages we need to fetch, then iterate over all pages with purrr::map_df to bind everything into a single tibble.

fetch_all_harvest_timesheets <- function(per_page = 100) {
  # Fetch the first page to get total page count
  first_page_response <- fetch_harvest_timesheet_page(page = 1, per_page = per_page)
  total_pages <- as.integer(headers(first_page_response)$`x-total-pages`)
  
  # If there's only one page, return it immediately
  if (total_pages == 1) {
    return(first_page_response)
  }
  
  # Iterate over remaining pages and bind all results
  remaining_pages <- 2:total_pages
  
  all_pages <- map_df(remaining_pages, ~fetch_harvest_timesheet_page(page = .x, per_page = per_page))
  
  bind_rows(first_page_response, all_pages)
}

4. Use the Function & Validate #

Now just call the function to get all your timesheet data in one tidy tibble:

all_timesheets <- fetch_all_harvest_timesheets()

# Quick check to confirm all data is there
glimpse(all_timesheets)

Key Notes for Readability & Maintainability #

• Modular Functions: Splitting into single-page and full-fetch functions makes it easy to debug or modify parts of the workflow without breaking everything.
• Tidyverse Principles: Using map_df automatically binds results into a tibble, and pluck makes parsing JSON responses clean and explicit.
• Error Handling: stop_for_status will throw a clear error if the API request fails, so you can catch issues early.
• Credentials: For production, consider storing your API token/account ID in environment variables (using Sys.getenv()) instead of hardcoding them—this is more secure and keeps sensitive data out of your codebase.

内容的提问来源于stack exchange，提问作者Mark Druffel