站点www版本存在隐藏robots.txt拦截根域，求.htaccess解决方案

Let's work through this problem step by step—first, let's figure out where that mysterious www.example.com/robots.txt is coming from, then fix it so crawlers use your actual, editable robots.txt.

First: Why is there a "ghost" robots.txt on www? #

That default User-agent: * Disallow: / content is almost always one of two things:

• Hosting/CDN default: Many web hosts or CDNs automatically serve this generic block-all robots.txt when they detect no actual robots.txt file exists at the requested URL. It's a safety measure to prevent accidental full-site crawling if you forget to add a robots.txt.
• Server config rule: Your web server (Apache/Nginx) might have a virtual host or global config rule that serves this preset content whenever www.example.com/robots.txt is requested and no file is found.

Solution 1: Create a real robots.txt for the www subdomain #

If your www and non-www domains point to separate root directories, just copy the content from example.com/robots.txt into a new robots.txt file in the www subdomain's root directory. This will override the default ghost content immediately.

If both domains share the same root directory, skip to the next solution—creating a duplicate file won't help here.

Solution 2: Force all robots.txt requests to use your editable file (via .htaccess) #

Since your current .htaccess already handles HTTPS redirects, we can add a rule to make sure any request for robots.txt (whether on www or non-www) uses your actual, editable file.

Option A: 301 Redirect (tells crawlers to use the non-www robots.txt)

Add this rule after your existing HTTPS redirect in .htaccess:

RewriteEngine On

# Existing HTTPS redirect (keep this)
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

# Redirect all robots.txt requests to your editable non-www version
RewriteRule ^robots.txt$ https://example.com/robots.txt [L,R=301]

Search engines will follow this 301 redirect and prioritize the content from your non-www robots.txt instead of the ghost one.

Option B: Internal Rewrite (URL stays the same, serves your real file)

If you don't want the URL to change (keep www.example.com/robots.txt in the address bar), use an internal rewrite. This works if both domains point to the same root directory:

RewriteEngine On

# Existing HTTPS redirect (keep this)
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

# Internally serve your real robots.txt for www requests
RewriteCond %{HTTP_HOST} ^www.example.com$
RewriteRule ^robots.txt$ /robots.txt [L]

This tells Apache to serve your existing robots.txt file whenever someone (or a crawler) requests www.example.com/robots.txt.

Bonus: Check CDN Caching #

If you're using a CDN, the ghost robots.txt might be cached. After making changes, clear the CDN's cache for the www.example.com/robots.txt path to ensure the new content is served immediately.

If All Else Fails #

Reach out to your web host's support team—ask if they have a default robots.txt configuration enabled for your www subdomain, and request to disable it or override it with your custom content.

内容的提问来源于stack exchange，提问作者Xione