如何配置带验证码的Content Locker并排除搜索引擎爬虫？

Got it, let's tackle this straightforwardly—you need a basic content locker that blocks regular visitors (to stop scraping) but lets search engine crawlers view everything, no fancy bells and whistles required. We'll focus on two easy implementations, starting with the core logic that makes this work.

The key here is checking the User-Agent (UA) string sent by the visitor. Search engines like Google, Bing, and DuckDuckGo send unique UA strings for their crawlers. We'll use this to decide whether to show the full content or the locker.

Important: Always combine front-end checks with backend validation—front-end only checks can be bypassed easily by scrapers, but backend UA checks are harder to fake for legitimate crawlers.

Common crawler UA strings to target:

• Googlebot (Google)
• Bingbot (Bing)
• DuckDuckBot (DuckDuckGo)
• Slurp (Yahoo)
• BaiduSpider (Baidu)

This is perfect if you want minimal friction for real visitors. We'll show a lock overlay unless the visitor is a crawler, then let them unlock with a single button click.

Backend Example (PHP) #

First, add this at the top of your page to check for crawlers:

<?php
// List of known crawler UA strings
$crawlers = ['Googlebot', 'Bingbot', 'DuckDuckBot', 'Slurp', 'BaiduSpider'];
$is_crawler = false;

// Check if the visitor's UA matches any crawler
foreach ($crawlers as $crawler) {
    if (strpos($_SERVER['HTTP_USER_AGENT'], $crawler) !== false) {
        $is_crawler = true;
        break;
    }
}
?>

Frontend HTML/JS #

Then render the content or locker based on the backend check:

<!-- Full Content (hidden by default unless crawler) -->
<div id="full-content" style="display: <?php echo $is_crawler ? 'block' : 'none'; ?>">
    <!-- Your actual website content goes here -->
    <h1>Your Protected Content</h1>
    <p>This is the stuff you don't want scraped!</p>
</div>

<!-- Content Locker (shown to non-crawlers) -->
<div id="content-locker" style="display: <?php echo $is_crawler ? 'none' : 'block'; ?>; position: fixed; top:0; left:0; width:100%; height:100%; background: #fff; display:flex; flex-direction:column; align-items:center; justify-content:center;">
    <h2>Please unlock to view content</h2>
    <button id="unlock-btn" style="padding: 12px 24px; font-size:16px; cursor:pointer;">Unlock Content</button>
</div>

<script>
// Unlock button logic
document.getElementById('unlock-btn').addEventListener('click', function() {
    document.getElementById('content-locker').style.display = 'none';
    document.getElementById('full-content').style.display = 'block';
});
</script>

If you want a bit more protection against automated scrapers, add a basic math captcha instead of a simple button.

Updated Frontend JS (Replace the button section above) #

<div id="content-locker" style="display: <?php echo $is_crawler ? 'none' : 'block'; ?>; position: fixed; top:0; left:0; width:100%; height:100%; background: #fff; display:flex; flex-direction:column; align-items:center; justify-content:center;">
    <h2>Solve this to unlock content</h2>
    <p id="captcha-question"></p>
    <input type="number" id="captcha-input" style="padding:8px; margin:10px 0;">
    <button id="submit-captcha" style="padding: 12px 24px; font-size:16px; cursor:pointer;">Submit</button>
</div>

<script>
// Generate random math question
const num1 = Math.floor(Math.random() * 10) + 1;
const num2 = Math.floor(Math.random() * 10) + 1;
document.getElementById('captcha-question').textContent = `${num1} + ${num2} = ?`;

// Captcha submit logic
document.getElementById('submit-captcha').addEventListener('click', function() {
    const userAnswer = parseInt(document.getElementById('captcha-input').value);
    if (userAnswer === num1 + num2) {
        document.getElementById('content-locker').style.display = 'none';
        document.getElementById('full-content').style.display = 'block';
    } else {
        alert('Wrong answer, please try again!');
    }
});
</script>

Pro Tips #

• Test the crawler detection: Simulate a crawler UA string in your browser's developer tools to verify that crawlers see the full content.
• Avoid overblocking: Add more crawler UA strings if you need to support other regional search engines.
• Don't go overboard: Since you want simplicity, skip complex captchas like reCAPTCHA—they'll needlessly annoy real visitors.

内容的提问来源于stack exchange，提问作者Andrei Ghițan