如何解决Facebook爬虫引发服务器高资源占用及robots.txt设置无效问题？

Hey Jake, let's tackle this issue you're having with Facebot hogging your server resources—first off, your robots.txt setup has a small but critical formatting error that's probably why the crawl delay isn't working. Let's walk through fixes and better solutions step by step.

First: Fix Your robots.txt Format #

Your current entry has a broken Disallow line—you need to explicitly allow crawling (even for all content) by setting Disallow: (with a trailing space, or leave it empty). Here's the corrected version:

User-agent: Facebot
Disallow:
Crawl-delay: 5

The problem with your original code is that Disallow: without a value is invalid in standard robots.txt syntax, so Facebot might be ignoring the entire block. Even with this fix though, keep in mind: not all crawlers strictly adhere to robots.txt, especially if they're misconfigured or malicious imitators.

Better Solutions to Block/Throttle Excessive Crawling #

Since robots.txt is more of a polite request than an enforcement tool, you'll want to add server-level safeguards:

1. Throttle Requests with Apache Modules #

You can use Apache's built-in modules to limit how often Facebot can hit your server:

• Using mod_rewrite to rate-limit:
  Add this to your Apache config or .htaccess file to restrict Facebot to, say, 10 requests per minute:

  RewriteEngine On
  # Track request time for Facebot
  RewriteCond %{HTTP_USER_AGENT} Facebot [NC]
  RewriteCond %{ENV:FACEBOT_LAST_REQUEST} !^$
  RewriteCond %{ENV:FACEBOT_LAST_REQUEST} >%{TIME_MIN}
  # Return 429 Too Many Requests if over limit
  RewriteRule ^ - [L,R=429]
  # Set the last request time if within limit
  RewriteCond %{HTTP_USER_AGENT} Facebot [NC]
  RewriteRule ^ - [E=FACEBOT_LAST_REQUEST:%{TIME_MIN}]
  

• Using mod_ratelimit:
  If you have this module enabled, it's a cleaner way to set limits:

  <IfModule mod_ratelimit.c>
      <Location "/">
          # Apply limit only to Facebot
          SetEnvIf User-Agent Facebot facebot_limit
          # Allow 10 requests per minute
          RATE_LIMIT 10 request/minute env=facebot_limit
      </Location>
  </IfModule>
  

2. Verify It's Actually Facebot #

Lots of malicious crawlers pretend to be legitimate ones like Facebot. Add a check to confirm the request comes from Facebook's official IP range:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Facebot [NC]
# Block if the request's reverse DNS doesn't end with facebook.com
RewriteCond %{REMOTE_HOST} !\.facebook\.com$
RewriteRule ^ - [L,R=403]

You can also cross-reference the IP with Facebook's published IP ranges to be extra strict.

3. Reduce MySQL Load Indirectly #

The spike in MySQL memory is likely because every crawler request is hitting your database. Fix this by:

• Caching static content: Use Apache's mod_cache to serve cached versions of pages, so crawlers don't trigger fresh database queries.
• Optimizing database queries: Make sure your SQL queries are indexed properly, and avoid unnecessary joins or heavy computations in requests that crawlers hit often.
• Tuning MySQL settings: Adjust innodb_buffer_pool_size to allocate memory efficiently (aim for 50-70% of your server's RAM if MySQL is the main database user).

4. Consider a Web Application Firewall (WAF) #

A WAF can automatically detect and block excessive crawling behavior, even from legitimate crawlers that ignore robots.txt. Many hosting providers include basic WAF tools, or you can set up open-source options like ModSecurity.

Final Notes #

Start with fixing the robots.txt as a quick win, then layer in the Apache rate-limiting and crawler verification. Pair that with database and caching optimizations to reduce the overall resource impact—this should get your CPU and memory usage back to normal.

内容的提问来源于stack exchange，提问作者Jake Jacobs