基于requests分页参数循环爬取单公司职位的异常与优化咨询

Hey there, let's figure out why your pagination loop is only grabbing 25 jobs instead of the full 488, and fix it without the high overhead of your current workaround.

First, Diagnose the Root Cause #

Your issue of getting 1 job per page (total 25) almost always boils down to one of these two problems:

• Incorrect pagination parameters: You might have accidentally set a per_page or limit value of 1 instead of the default 20. Or you're using the wrong parameter names (some APIs use offset instead of page for pagination).
• Broken loop logic: Your loop might be stopping too early (e.g., hardcoding 25 iterations instead of calculating total pages) or not incrementing the page/offset correctly between requests.

Step-by-Step Fix #

Let's walk through a reliable, efficient approach to scrape all 488 jobs:

1. Confirm the API's Pagination Rules

First, manually test the API to understand how it handles pagination. Send a test request for the first page and check the response (usually JSON):

import requests

base_url = "YOUR_API_ENDPOINT_HERE"
params = {
    "company_id": "TARGET_COMPANY_ID",  # Replace with your company's ID
    "page": 0,  # Or offset=0 if the API uses offset
    "per_page": 20  # Use the API's default max allowed value to minimize requests
}

response = requests.get(base_url, params=params)
response.raise_for_status()  # Catch HTTP errors early
data = response.json()
print(data)

Look for fields like:

• total: Total number of jobs (should be 488)
• per_page: Number of jobs returned per page
• current_page or offset: The current position in the dataset

2. Build a Correct Pagination Loop

Based on the API's rules, use one of these two loop patterns:

Pattern 1: Using Page Numbers

If the API uses page and per_page parameters:

from math import ceil
import requests
import time

base_url = "YOUR_API_ENDPOINT_HERE"
company_id = "TARGET_COMPANY_ID"
per_page = 20
all_jobs = []

# Fetch first page to get total jobs
first_response = requests.get(base_url, params={"company_id": company_id, "page": 0, "per_page": per_page})
first_response.raise_for_status()
first_data = first_response.json()
total_jobs = first_data["total"]
total_pages = ceil(total_jobs / per_page)

# Loop through all pages
for page in range(total_pages):
    # Add a small delay to avoid triggering anti-scraping limits
    time.sleep(1)
    
    response = requests.get(base_url, params={
        "company_id": company_id,
        "page": page,
        "per_page": per_page
    })
    response.raise_for_status()
    page_data = response.json()
    all_jobs.extend(page_data["jobs"])  # Replace "jobs" with the actual key in your response

print(f"Successfully scraped {len(all_jobs)} jobs!")

Pattern 2: Using Offset & Limit

If the API uses offset (starting position) and limit (jobs per page):

import requests
import time

base_url = "YOUR_API_ENDPOINT_HERE"
company_id = "TARGET_COMPANY_ID"
limit = 20
offset = 0
all_jobs = []

while True:
    time.sleep(1)
    
    response = requests.get(base_url, params={
        "company_id": company_id,
        "offset": offset,
        "limit": limit
    })
    response.raise_for_status()
    page_data = response.json()
    jobs = page_data["jobs"]
    
    # Stop loop when no more jobs are returned
    if not jobs:
        break
    
    all_jobs.extend(jobs)
    offset += limit

print(f"Successfully scraped {len(all_jobs)} jobs!")

3. Optimize to Avoid High Overhead

To fix your "high overhead, lots of duplicates" issue:

• Use a session: Reuse HTTP connections with requests.Session() to reduce latency:

  session = requests.Session()
  # Replace all requests.get() calls with session.get()
  

• Skip duplicate requests: Track which pages/offsets you've already fetched to avoid re-requesting them.
• Add error handling: Use try-except blocks to handle temporary failures without restarting the entire scrape:

  for page in range(total_pages):
      try:
          response = session.get(base_url, params=...)
          response.raise_for_status()
          # Process data
      except requests.exceptions.RequestException as e:
          print(f"Failed to fetch page {page}: {e}. Retrying...")
          time.sleep(5)  # Wait longer before retrying
  

Why Your Previous Loop Failed #

Chances are you either:

• Set per_page=1 instead of 20, leading to 1 job per request, and your loop stopped after 25 iterations.
• Didn't calculate the total number of pages correctly, so you didn't loop through all required pages to reach 488 jobs.

内容的提问来源于stack exchange，提问作者Edward