如何为获取外部CSS的HTTP请求设置自定义Referer请求头

Got it, this is a classic case of server-side Referer validation by the target site (www.x.com). Their server is checking that requests for style.css come from their own domain, so we need to explicitly set the correct Referer header when our server fetches that resource. Let's break down solutions for common server environments:

1. Node.js (Express with Axios or HTTP Module) #

If you're using Node.js to handle the request, you can set the Referer header directly in your HTTP client configuration:

Using Axios: #

const axios = require('axios');

async function fetchExternalCSS(req, res) {
  try {
    const response = await axios.get('http://www.x.com/style.css', {
      headers: {
        'Referer': 'http://www.x.com', // Critical: matches the target domain
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' // Optional but recommended to mimic a browser
      }
    });
    // Send the CSS content to your client
    res.set('Content-Type', 'text/css');
    res.send(response.data);
  } catch (error) {
    console.error('Failed to fetch CSS:', error);
    res.sendStatus(500);
  }
}

// Route to serve the proxied CSS
app.get('/proxied-style.css', fetchExternalCSS);

Using Native HTTP Module: #

const http = require('http');

app.get('/proxied-style.css', (req, res) => {
  const options = {
    hostname: 'www.x.com',
    path: '/style.css',
    headers: {
      'Referer': 'http://www.x.com',
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    }
  };

  const proxyReq = http.get(options, (proxyRes) => {
    res.setHeader('Content-Type', proxyRes.headers['content-type']);
    proxyRes.pipe(res);
  });

  proxyReq.on('error', (err) => {
    console.error('Proxy request error:', err);
    res.sendStatus(500);
  });
});

2. Nginx Reverse Proxy #

If you're using Nginx to proxy the CSS request to www.x.com, add the proxy_set_header directive to your location block:

server {
  # ... your existing server config ...

  location /proxied-style.css {
    proxy_pass http://www.x.com/style.css;
    proxy_set_header Referer http://www.x.com; # Set the required Referer
    proxy_set_header User-Agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36";
    proxy_set_header Host www.x.com; # Ensure Host header matches target domain (often required)
  }
}

3. PHP with cURL #

For PHP-based servers, use cURL's CURLOPT_REFERER option:

<?php
$cssUrl = 'http://www.x.com/style.css';
$referer = 'http://www.x.com';

$ch = curl_init($cssUrl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36');

$cssContent = curl_exec($ch);
if (curl_errno($ch)) {
  http_response_code(500);
  echo 'Failed to fetch CSS: ' . curl_error($ch);
} else {
  header('Content-Type: text/css');
  echo $cssContent;
}
curl_close($ch);
?>

Important Notes #

• Respect Target Site Policies: Make sure you're allowed to fetch and proxy their resources. Check their robots.txt and terms of service to avoid legal issues.
• Additional Headers: Some sites may also validate the Host or User-Agent headers, so including those (as shown in examples) can help avoid 403s.
• Caching: Consider adding caching for the fetched CSS to reduce repeated requests to the target server (good for performance and to avoid hitting rate limits).

内容的提问来源于stack exchange，提问作者xamcx