基于Scrapy Splash实现JS渲染页面翻页爬取的技术咨询

Hey Kevin, great job getting JS-rendered pages scraped already—stepping into pagination with Splash is totally doable, and I’ll walk you through the key steps and examples to make this work smoothly.

Core Idea #

Splash uses Lua scripts to simulate real browser interactions, so clicking the "Next" button boils down to:

1. Locating the button element in the page
2. Triggering a click event
3. Waiting for the new page to render
4. Repeating until there’s no "Next" button left

Step-by-Step Implementation #

1. Identify the "Next" Button Selector

First, use your browser’s dev tools to find a reliable selector for the Next button. Common options are:

• CSS selector: button.next-page, a[rel="next"]
• XPath: //button[contains(text(), 'Next')]

Test the selector in your browser’s console with document.querySelector("your-selector") to make sure it targets the right element.

2. Write a Lua Script with Loop Logic

This script will handle the pagination loop, data extraction, and button clicks. Here’s a practical example:

function main(splash, args)
  -- Set a realistic user-agent to avoid detection
  splash:set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36')
  
  -- Load the initial page
  splash:go(args.url)
  splash:wait(2) -- Wait for initial content to render
  
  local all_results = {}

  while true do
    -- Extract data from the current page (customize this to your needs)
    local current_page_data = splash:evaljs([[
      Array.from(document.querySelectorAll('.content-item')).map(item => ({
        title: item.querySelector('.item-title').textContent.trim(),
        description: item.querySelector('.item-desc').textContent.trim()
      }))
    ]])
    -- Add current page's data to the results list
    for _, item in ipairs(current_page_data) do
      table.insert(all_results, item)
    end

    -- Check if the Next button exists and is clickable
    local has_next_button = splash:evaljs([[
      const nextBtn = document.querySelector('button.next');
      nextBtn && !nextBtn.disabled;
    ]])

    if not has_next_button then
      break -- Exit loop if no valid Next button
    end

    -- Click the Next button
    splash:evaljs([[
      document.querySelector('button.next').click();
    ]])

    -- Wait for the new page to load (use wait_for_element for better reliability)
    -- Option 1: Fixed wait time
    splash:wait(3)
    -- Option 2: Wait for a unique element on the next page (more robust)
    -- splash:wait_for_element('.new-page-identifier', 10)
  end

  return {
    results = all_results,
    html = splash:html() -- Optional: Return final page HTML for debugging
  }
end

3. Integrate with Your Python Code

Call the Splash service from Python using requests to run the Lua script and retrieve the results:

import requests

# Replace with your Splash server URL (default is localhost:8050)
SPLASH_SERVER = "http://localhost:8050/execute"

def scrape_paginated_pages(target_url):
    # Load the Lua script (you can also read it from a file for cleaner code)
    lua_script = """
    function main(splash, args)
      -- Paste the full Lua script from above here
    end
    """

    payload = {
        "lua_source": lua_script,
        "url": target_url
    }

    response = requests.post(SPLASH_SERVER, json=payload)
    response.raise_for_status() -- Raise error if request fails
    return response.json()["results"]

# Example usage
if __name__ == "__main__":
    target_url = "https://example.com/your-target-paginated-page"
    all_scraped_data = scrape_paginated_pages(target_url)
    print(f"Scraped {len(all_scraped_data)} items across all pages!")

Key Tips for Reliability #

• Handle Anti-Scraping Measures: Add cookies, rotate user-agents, or introduce random delays between clicks with splash:math.random() to mimic human behavior.
• Avoid Infinite Loops: Add a maximum loop count (e.g., local loop_count = 0; while loop_count < 20 do ... loop_count = loop_count +1 end) to prevent getting stuck.
• Debug with Splash UI: Use the Splash web interface (http://localhost:8050) to test your Lua script step-by-step—you can see the rendered page and debug errors easily.
• Check for Button State: Some sites disable the Next button on the last page instead of hiding it. Always check if the button is enabled before clicking.

内容的提问来源于stack exchange，提问作者kevin