无URL分页参数的多页表格Python爬取方案问询

Great question! When you're dealing with sites that don't expose page numbers in the URL, it almost always means the content is loaded dynamically—either when you click a "Next" button (triggering an AJAX request) or as you scroll down the page. Below are two reliable methods to scrape all those pages using Python, with practical code examples:

If the site relies on user actions (like clicking a "Next Page" button) to load new content, Selenium is a great tool—it mimics real browser behavior, so you can automate those clicks and scrape data as you go.

Steps to implement: #

1. Install Selenium and download the matching browser driver (e.g., ChromeDriver for Chrome):

   pip install selenium
   

2. Write code to automate the process:

   from selenium import webdriver
   from selenium.webdriver.common.by import By
   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   from selenium.common.exceptions import NoSuchElementException
   import time
   
   # Initialize the browser (Chrome in this example)
   driver = webdriver.Chrome()
   driver.get("YOUR_TARGET_URL")
   
   all_data = []
   
   while True:
       # Scrape data from the current page (adjust selector to match your table)
       table_rows = driver.find_elements(By.CSS_SELECTOR, "table tr")
       for row in table_rows[1:]:  # Skip header row if needed
           row_data = [cell.text for cell in row.find_elements(By.TAG_NAME, "td")]
           if row_data:
               all_data.append(row_data)
   
       # Try to click the "Next" button
       try:
           next_button = WebDriverWait(driver, 10).until(
               EC.element_to_be_clickable((By.CSS_SELECTOR, "button.next-page"))  # Adjust selector to match the site
           )
           next_button.click()
           time.sleep(2)  # Give time for the page to load (adjust based on site speed)
       except NoSuchElementException:
           # No more pages—exit loop
           print("No more pages to scrape.")
           break
   
   # Close the browser
   driver.quit()
   
   # Process or save your data
   print(f"Scraped {len(all_data)} rows of data.")
   

   Note: You'll need to update the CSS selectors for the table rows and "Next" button to match the actual site structure. Use your browser's dev tools to inspect elements and get the right selectors.

Many sites load dynamic content via background AJAX requests. Instead of simulating a browser, you can directly call these API endpoints with the requests library—this is faster and uses fewer resources.

Steps to implement: #

1. Use your browser's Developer Tools (F12) to find the AJAX request:
   • Go to the "Network" tab, then click "XHR" to filter requests.
   • Click the "Next" button on the site, and look for the request that loads the next page of data.
   • Check the request URL, method (GET/POST), and any parameters (like offset, limit, or a cursor token that tracks your position).
2. Write code to loop through these requests:

   import requests
   from bs4 import BeautifulSoup  # If the response is HTML; use json() if it's JSON
   
   base_url = "https://example.com/api/load-table-data"
   offset = 0
   limit = 20  # Adjust based on how many rows are loaded per page
   all_data = []
   
   while True:
       # Send request with parameters (adjust based on what you found in dev tools)
       params = {
           "offset": offset,
           "limit": limit
       }
       response = requests.get(base_url, params=params, headers={"User-Agent": "Mozilla/5.0"})
       response.raise_for_status()  # Raise error if request fails
   
       # Parse response (example for HTML; if JSON, use response.json())
       soup = BeautifulSoup(response.text, "html.parser")
       table_rows = soup.select("table tr")
       if not table_rows[1:]:  # No more data
           break
   
       # Extract data
       for row in table_rows[1:]:
           row_data = [cell.get_text(strip=True) for cell in row.select("td")]
           all_data.append(row_data)
   
       # Update offset for next page
       offset += limit
       # If using a cursor instead of offset, extract the next cursor from the response
       # e.g., next_cursor = response.json().get("next_cursor")
       # if not next_cursor: break
   
   print(f"Scraped {len(all_data)} rows of data.")
   

   Pro tip: Always include a User-Agent header to avoid being blocked. Some sites might require additional headers (like Referer or Authorization)—copy these from the request in dev tools.

Important Notes: #

• Respect the site's rules: Check robots.txt and the site's Terms of Service before scraping. Avoid sending too many requests too quickly—add delays (time.sleep()) or use a rate limiter.
• Handle anti-scraping measures: Some sites use CAPTCHAs or IP blocking. If you hit these, consider using proxies or rotating user agents.
• Test incrementally: Start with scraping a single page first, then add the loop for multiple pages.

内容的提问来源于stack exchange，提问作者Demetri Pananos