Laravel应用付费外链遭爬虫恶意访问，求访问限制方案

Hey there, let's fix this crawler-induced bill shock once and for all. Unwanted bots hitting your external paid links can quickly rack up costs, but there are several solid strategies to block them and protect your budget in Laravel:

The first line of defense is ensuring only subscribed, authenticated users can even reach the pages or routes that lead to your paid external links.

• Add Laravel's built-in auth middleware to your paid content routes to block unauthenticated visitors entirely:

  // routes/web.php
  Route::get('/premium-content', [PremiumContentController::class, 'show'])
      ->middleware('auth', 'subscribed');
  

• Create a custom subscribed middleware to verify the user has an active paid subscription:

  // app/Http/Middleware/CheckSubscriptionStatus.php
  public function handle(Request $request, Closure $next)
  {
      if (!auth()->user()?->hasActiveSubscription()) {
          abort(403, 'Access to paid content requires an active subscription.');
      }
  
      return $next($request);
  }
  

Don't forget to register the middleware in app/Http/Kernel.php.

Most crawlers identify themselves with a unique User-Agent string. You can build a middleware to block non-human traffic from accessing your paid routes.

• Create a BlockUnwantedCrawlers middleware:

  // app/Http/Middleware/BlockUnwantedCrawlers.php
  public function handle(Request $request, Closure $next)
  {
      $blockedAgents = [
          'Googlebot', 'Bingbot', 'Slurp', 'DuckDuckBot',
          'AhrefsBot', 'SemrushBot', 'MJ12bot', 'YandexBot'
          // Add any other crawlers you want to block
      ];
  
      $userAgent = strtolower($request->header('User-Agent'));
  
      foreach ($blockedAgents as $agent) {
          if (str_contains($userAgent, strtolower($agent))) {
              abort(403, 'Crawlers are not allowed to access this content.');
          }
      }
  
      return $next($request);
  }
  

Attach this middleware to your paid content routes to keep bots out. Note: If you want search engines to index your public pages but not paid ones, make sure to only apply this middleware to the premium routes.

Even if some bots slip through, rate limiting can cap the number of requests they can make, preventing sky-high bills. Laravel makes this easy with the throttle middleware:

// routes/web.php
Route::post('/trigger-paid-request', [PremiumContentController::class, 'triggerExternalRequest'])
    ->middleware('auth', 'subscribed', 'throttle:10,1'); // 10 requests per minute

You can customize the limits based on your needs—for example, throttle:5,60 allows 5 requests per hour for each user.

Instead of putting raw external paid URLs in your frontend (where crawlers can scrape and hit them directly), proxy the request through your Laravel backend. This way, all external calls go through your server only after verifying the user's permissions:

// app/Http/Controllers/PremiumContentController.php
public function triggerExternalRequest(Request $request)
{
    // Verify user is subscribed (already handled by middleware, but double-check if needed)
    if (!auth()->user()->hasActiveSubscription()) {
        abort(403);
    }

    // Make the request to the external paid service from your backend
    $response = Http::get('https://external-paid-service.com/api/content', [
        'user_id' => auth()->id(),
        // Other required parameters
    ]);

    // Return the content to the user
    return response()->json($response->json());
}

Now, crawlers can't directly hit the external service—they have to go through your authenticated, rate-limited endpoint first.

While not 100% foolproof (malicious bots ignore it), robots.txt tells compliant crawlers to stay away from your paid content routes. Add this to your public robots.txt file:

User-agent: *
Disallow: /premium-content/
Disallow: /trigger-paid-request/

If you have pages where users initiate paid requests, add a CAPTCHA to ensure the requester is human. Laravel integrates smoothly with reCAPTCHA:

• Install the package and configure it
• Add the CAPTCHA field to your form:

  <div class="g-recaptcha" data-sitekey="{{ config('services.recaptcha.site_key') }}"></div>
  

• Verify the response in your controller:

  $response = Http::post('https://www.google.com/recaptcha/api/siteverify', [
      'secret' => config('services.recaptcha.secret_key'),
      'response' => $request->input('g-recaptcha-response'),
  ]);
  
  if (!$response->json('success')) {
      return back()->withErrors(['captcha' => 'Please verify you are human.']);
  }
  

Combining these strategies should drastically reduce unwanted crawler traffic to your paid links. Start with authentication/authorization and proxying external requests—those are the most reliable ways to prevent unauthorized charges.

内容的提问来源于stack exchange，提问作者Tim