Scrapinghub中Scrapy爬虫报“TCP connection timed out”本地正常，求助

Hey there, let's dig into why your Scrapy crawler runs smoothly on your local machine but throws errors on Scrapinghub—especially since you're mixing requests and BeautifulSoup with Scrapy's framework. The truncated traceback you shared points to Twisted's deferred system, which tells us the issue is likely tied to environment differences or conflicts between synchronous (requests) and asynchronous (Scrapy/Twisted) code.

Here are the most common fixes and troubleshooting steps:

1. Fix Dependency Version Mismatches #

Scrapinghub runs Python 3.6 (from the traceback path /usr/local/lib/python3.6/), which might be older than your local Python version. Discrepancies in requests, beautifulsoup4, or scrapy versions can cause unexpected behavior.

• Solution: Create a requirements.txt file in your project root with the exact versions you tested locally, like:

  scrapy==2.5.1
  requests==2.26.0
  beautifulsoup4==4.10.0
  lxml==4.6.3  # If you use lxml for parsing
  

  Then, in your Scrapinghub project settings, configure it to install dependencies from this file.

2. Resolve Async/Sync Code Conflicts #

Scrapy is built on Twisted's asynchronous event loop, but requests is a synchronous library. Calling requests.get() directly in a Scrapy callback blocks the event loop, which works fine locally but can trigger errors in Scrapinghub's distributed environment.

• Solution 1 (Recommended): Switch to Scrapy's native scrapy.Request instead of requests—it’s designed to work with Scrapy’s async model:

  def parse(self, response):
      yield scrapy.Request(
          url="https://target-url.com",
          callback=self.parse_with_soup,
          headers={"User-Agent": "Your-Local-User-Agent"}
      )
  
  def parse_with_soup(self, response):
      soup = BeautifulSoup(response.text, 'html.parser')
      # Process your soup data here
  

• Solution 2 (If You Must Use Requests): Wrap requests calls in Twisted’s thread pool to avoid blocking the event loop:

  from twisted.internet.threads import deferToThread
  import requests
  from bs4 import BeautifulSoup
  
  def parse(self, response):
      # Delegate the synchronous request to a separate thread
      yield deferToThread(self.fetch_with_requests, "https://target-url.com")
  
  def fetch_with_requests(self, url):
      try:
          resp = requests.get(url, timeout=10, headers={"User-Agent": "Your-Local-User-Agent"})
          resp.raise_for_status()  # Raise error for 4xx/5xx status codes
          soup = BeautifulSoup(resp.text, 'html.parser')
          # Return processed items or data
          return {"data": soup.find("div", class_="content").text}
      except Exception as e:
          self.logger.error(f"Failed to fetch {url}: {str(e)}")
          raise  # Re-raise to let Scrapy handle the failure
  

3. Check Network/IP Restrictions #

Scrapinghub’s server IPs might be blocked by the target website, while your local IP isn’t. This would cause requests to fail silently or throw errors that bubble up to Twisted’s defer system.

• Solution:
  • Check Scrapinghub’s full logs for requests error details (like 403 Forbidden or 503 Service Unavailable).
  • Match your local request headers (especially User-Agent, Referer) in your requests calls to avoid being flagged as a bot.
  • If IP blocking is confirmed, set up a proxy in your requests calls or use Scrapy’s proxy middleware.

4. Capture Unhandled Exceptions #

Local runs might have silent failures you didn’t notice, while Scrapinghub’s environment exposes uncaught exceptions. The truncated traceback hides the root cause—you need more context.

• Solution: Add detailed exception logging around your requests and BeautifulSoup code, then check Scrapinghub’s full logs:

  def fetch_with_requests(self, url):
      try:
          resp = requests.get(url)
          self.logger.info(f"Got response {resp.status_code} for {url}")
          soup = BeautifulSoup(resp.text, 'html.parser')
          # Your parsing logic here
      except requests.exceptions.RequestException as e:
          self.logger.error(f"Request error: {str(e)} | URL: {url}")
          raise
      except Exception as e:
          self.logger.error(f"Parsing error: {str(e)} | URL: {url}")
          raise
  

5. Optimize for Scrapinghub’s Resource Limits #

Scrapinghub’s crawler instances have memory/CPU limits. If your code processes large pages or sends too many concurrent requests calls, it might hit these limits and crash.

• Solution:
  • Lower Scrapy’s CONCURRENT_REQUESTS setting in settings.py to reduce load.
  • Use lxml instead of html.parser for BeautifulSoup (it’s faster and uses less memory).
  • Avoid loading entire large pages into memory—process data incrementally if possible.

First step: Go check Scrapinghub’s full error logs (not just the truncated traceback) — it’ll show you the exact root cause, whether it’s a network error, missing dependency, or code conflict.

内容的提问来源于stack exchange，提问作者Krishna joshi