使用Urllibopener提取URL时触发HTTP错误，Craigslist爬虫脚本求助

Hey there! Let's work through why your extract_post_url function is throwing errors while scraping Craigslist for Mazda Miatas. Based on the partial code you shared and common headaches with Craigslist + Selenium, here are the most likely fixes to get your script back on track:

1. Fix Incomplete Imports (Your Code Cuts Off!) #

Your snippet ends with fro...—it looks like you're missing the critical By import needed to locate elements. Without this, Selenium can't find elements by CSS selector, XPath, etc., which will break your extract_post_url function immediately. Add this line to your imports:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By  # This is the line you're missing!

2. Beat Craigslist's Anti-Scraping & Dynamic Content #

Craigslist actively blocks automated tools like Selenium and loads content dynamically, which can cause "element not found" errors. Try these tweaks:

• Mask your Selenium fingerprint to avoid being detected:

  options = webdriver.ChromeOptions()
  options.add_argument("--headless=new")  # Run in background (optional but helpful)
  options.add_argument("--disable-blink-features=AutomationControlled")  # Hide automation flag
  options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")  # Mimic real browser
  
  driver = webdriver.Chrome(options=options)
  

• Wait for content to load before scraping in your extract_post_url function. Don't try to grab elements immediately after driver.get()—use WebDriverWait to ensure the post list is fully loaded:

  def extract_post_url(driver, search_url):
      driver.get(search_url)
      # Wait up to 10 seconds for post rows to appear
      WebDriverWait(driver, 10).until(
          EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".result-row"))
      )
      
      post_urls = []
      # Loop through each post row and extract the link
      for row in driver.find_elements(By.CSS_SELECTOR, ".result-row"):
          title_link = row.find_element(By.CSS_SELECTOR, ".result-title.hdrlnk")
          post_urls.append(title_link.get_attribute("href"))
      
      return post_urls
  

3. Update Your Element Selectors #

Craigslist changes its HTML structure occasionally—your tutorial's selectors might be outdated. To check:

1. Open your Mazda Miata Craigslist search page in a browser
2. Right-click a post title and select "Inspect"
3. Verify the CSS selector for the title link matches .result-title.hdrlnk (or update it to whatever the current selector is)

4. Avoid Rate Limiting #

Craigslist will block you if you send too many requests too fast. Add random delays between actions to mimic human behavior:

import time
import random

def extract_post_url(driver, search_url):
    driver.get(search_url)
    WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".result-row"))
    )
    
    time.sleep(random.uniform(2, 5))  # Wait 2-5 seconds randomly
    
    post_urls = []
    for row in driver.find_elements(By.CSS_SELECTOR, ".result-row"):
        title_link = row.find_element(By.CSS_SELECTOR, ".result-title.hdrlnk")
        post_urls.append(title_link.get_attribute("href"))
        time.sleep(random.uniform(0.5, 1.5))  # Small delay between extracting each link
    
    return post_urls

内容的提问来源于stack exchange，提问作者Amateur Python Learner