网页爬取问题：提取学校链接及表格均返回空DataFrame

Hey there! Let's break down why you're hitting empty DataFrames when scraping school links and their tables, even after trying CSS selectors and multiple XPaths. I'll walk through the most common issues and actionable fixes step by step.

Common Causes & Solutions #

1. Dynamic Content Loading (The #1 Culprit) #

Many modern websites load content like links or tables dynamically with JavaScript—meaning the initial HTML you get from requests won't include these elements.

• How to check: Right-click the page → "View Page Source" (not the "Elements" tab in dev tools), then Ctrl+F to search for a school name or table value you know should exist. If it doesn't show up, the content is loaded after the initial page load.
• Fix: Use a tool that renders JavaScript, like Selenium, Playwright, or Scrapy with Splash. For example, here's a quick Selenium snippet to wait for elements to load before scraping:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import pandas as pd

# Initialize driver (make sure you have ChromeDriver installed)
driver = webdriver.Chrome()
driver.get("your_target_page_url")

# Wait 10 seconds for school links to load
school_links = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class, 'school-card')]//a"))
)
link_list = [link.get_attribute("href") for link in school_links]

# Scrape tables from each school page
all_tables = []
for link in link_list:
    driver.get(link)
    # Wait for the table to load
    table_element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "table.school-details-table"))
    )
    # Convert table to DataFrame
    df = pd.read_html(table_element.get_attribute("outerHTML"))[0]
    all_tables.append(df)

driver.quit()
# Combine all results
final_df = pd.concat(all_tables, ignore_index=True)

2. Incorrect Selector Targeting #

Even if content is static, your selectors might be missing the mark due to:

• Dynamic class/id names: Some sites generate random class names on each load (e.g., school-link-abc123). Use stable attributes instead—like text content, parent elements with fixed classes, or XPath functions like contains():

  # Instead of //a[@class='random-class-123']
  //div[@class='school-list-container']//a[contains(text(), 'High School')]
  

• Iframes: If the table is nested inside an <iframe>, you'll need to switch to the iframe first before selecting elements:

  # Switch to iframe using its ID or index
  driver.switch_to.frame("school-iframe")
  # Now scrape the table
  # Switch back to main content when done
  driver.switch_to.default_content()
  

• Nested elements: Double-check if your selector is targeting the right parent/child hierarchy. Use the dev tools' "Inspect" feature to right-click the element → "Copy" → "Copy XPath" or "Copy CSS selector" to get a working base selector.

3. Anti-Scraping Measures #

Sites might block your requests or return empty content if they detect automated tools:

• Add a realistic User-Agent: Many sites block the default requests User-Agent. Add a browser-like header:

  import requests
  headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
  }
  response = requests.get(url, headers=headers)
  

• Check response status & content: Print response.status_code (should be 200) and response.text[:500] to see if you're getting the actual page, not a captcha or blocked message.
• Use sessions: If the site requires cookies (e.g., after a welcome banner), use requests.Session() to persist cookies across requests.

4. Flawed Scraping Logic #

• Missing error handling: If a school link returns a 404 error, your code might skip it silently instead of logging the issue. Add try/except blocks to catch errors:

  for link in link_list:
      try:
          driver.get(link)
          # scrape logic here
      except Exception as e:
          print(f"Failed to scrape {link}: {str(e)}")
          continue
  

• Incorrect table parsing: If using pd.read_html(), make sure the page has a valid <table> element. Sometimes tables are built with <div>s instead—you'll need to extract rows/columns manually in that case.

Final Tip #

If you can share the specific target URL (assuming it's public), it'll be much easier to pinpoint the exact issue. But even without it, working through these steps should help you track down why your DataFrames are coming up empty.

内容的提问来源于stack exchange，提问作者Tdebeus