使用Python Selenium爬取Craigslist时遭遇socket.error 61问题求助

First off, socket.error 61 almost always means Craigslist is actively refusing your connection—let’s break down targeted fixes based on what you’ve already tried:

• Fix your browser automation fingerprint: Even with a new proxy, Chromedriver/PhantomJS leave obvious automated tool markers. For Chrome, tweak your driver setup to mimic a real user:

  from selenium import webdriver
  options = webdriver.ChromeOptions()
  options.add_argument('--headless=new')  # Use the modern headless mode
  options.add_argument('--disable-blink-features=AutomationControlled')
  options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36')
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options)
  

  PhantomJS is severely outdated—stick with Chrome or Firefox for better compatibility and easier fingerprint spoofing.

• Double-check your proxy is actually working: Sometimes proxies look configured but don’t route traffic correctly. Add a quick test: load https://api.ipify.org and print the page source to confirm your IP matches the proxy’s. If not, verify proxy authentication (if needed) and ensure you’re passing proxy arguments correctly to the driver.

• Slow down your request rate: Craigslist’s anti-crawl is aggressive. Even with a proxy, hitting the site too fast will get you blocked. Add random 2-5 second delays between requests, and implement exponential backoff for retries if connections fail.

• Rule out local network blocks: Your firewall, antivirus, or ISP might be blocking outgoing connections to Craigslist. Test your script on a different network (like a mobile hotspot) to eliminate this possibility.

• Inspect the request flow: Use Chrome DevTools or Wireshark to confirm your requests are even reaching Craigslist. If the connection is refused at the TCP level, your proxy/IP range might be fully blacklisted—try a new proxy provider if you haven’t already.

If you can share your full stack trace and a snippet of your scraping code, we can narrow this down even further!

内容的提问来源于stack exchange，提问作者Jake