如何遍历指定链接中的PDF文件数组并批量提取文本

Alright, let's walk through how to batch download and extract text from those SUGI 30 PDFs starting from Paper 085-30. Here's a step-by-step solution using R:

Step 1: Set up required packages #

We'll use pdftools for PDF text extraction, and rvest to scrape the target table of contents page for PDF links.

# Install packages if they're not already on your system
if (!require("pdftools")) install.packages("pdftools")
if (!require("rvest")) install.packages("rvest")

# Load the packages
library(pdftools)
library(rvest)

Step 2: Scrape the TOC page to get target PDF links #

First, we'll pull the HTML content from the TOC page, extract all PDF links, then filter to keep only those starting from Paper 085-30.

# URL of the TOC page
toc_url <- "http://www2.sas.com/proceedings/sugi30/toc.html#dp"

# Read the page content
toc_page <- read_html(toc_url)

# Extract all paper entries to find our starting point
paper_entries <- toc_page %>%
  html_elements("tr") %>%  # Papers are organized in table rows on this page
  html_text2() %>%
  str_trim()

# Extract all PDF links and convert relative paths to full URLs
pdf_links <- toc_page %>%
  html_elements("a[href$='.pdf']") %>%
  html_attr("href") %>%
  paste0("http://www2.sas.com", .)

# Find the index where Paper 085-30 starts
start_index <- which(grepl("Paper 085-30:", paper_entries))

# Filter links to keep only from our starting paper onward
target_pdf_links <- pdf_links[start_index:length(pdf_links)]

Step 3: Batch process PDFs (download + extract text) #

We'll loop through each target link, download the PDF, extract its text, and optionally save both the PDF and text files for later use.

# Create a folder to store PDFs and extracted text (keeps things organized)
if (!dir.exists("sugi30_pdfs")) dir.create("sugi30_pdfs")

# A list to store all extracted text (easy to access later)
all_extracted_text <- list()

# Loop through each link
for (link_num in seq_along(target_pdf_links)) {
  current_link <- target_pdf_links[link_num]
  pdf_filename <- gsub(".*/", "", current_link)  # Grab just the filename from the URL
  
  # Download the PDF (mode="wb" is critical for binary files like PDFs)
  download.file(current_link, paste0("sugi30_pdfs/", pdf_filename), mode = "wb", quiet = TRUE)
  
  # Extract text from the downloaded PDF
  pdf_text_content <- pdf_text(paste0("sugi30_pdfs/", pdf_filename))
  
  # Store the text in our list (using the PDF filename as the list item name)
  all_extracted_text[[pdf_filename]] <- pdf_text_content
  
  # Optional: Save the extracted text to a .txt file
  writeLines(pdf_text_content, paste0("sugi30_pdfs/", gsub(".pdf", ".txt", pdf_filename)))
  
  # Print progress so you know what's happening
  cat(paste("Finished processing:", pdf_filename, "\n"))
}

# Access extracted text example: all_extracted_text[["085-30.pdf"]]

Quick Notes #

• Double-check the website's terms of service to ensure you have permission to scrape and download these PDFs.
• If the page structure ever changes, you might need to adjust the CSS selector (like "tr")—use your browser's inspect tool to find the right elements containing paper entries.
• Remove quiet = TRUE from download.file() if you want to see detailed download progress for each file.

内容的提问来源于stack exchange，提问作者ASH