Puppeteer+Node.js爬虫在无头模式下被Adidas网站反爬检测的解决求助

我最近做了一个基于Puppeteer和Node.js的Adidas商品监控爬虫，本地用非无头模式（headless: false）的时候一切正常，但改成无头模式（现在用的是headless: 'new'）就会被网站检测到，完全没法正常访问页面。因为最终要部署在Railway平台上，必须用无头模式才行，所以想请教大家有没有解决办法。

我已经尝试了这些反检测手段：

• 集成了puppeteer-extra-plugin-stealth插件
• 用puppeteer-extra-plugin-anonymize-ua处理用户代理
• 用fingerprint-generator生成真实的浏览器指纹
• 配置了付费代理（WebShare的代理，带身份验证）
• 加了常见的启动参数（--no-sandbox、--disable-setuid-sandbox等）

以下是我的核心代码片段：

const puppeteer = require('puppeteer');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const AnonymizeUA = require('puppeteer-extra-plugin-anonymize-ua');
const { FingerprintGenerator } = require('fingerprint-generator');

puppeteer.use(StealthPlugin());
puppeteer.use(AnonymizeUA());

const fingerprintGenerator = new FingerprintGenerator({
  devices: ['desktop'],
  browsers: [{ name: 'chrome', minVersion: 114 }],
  operatingSystems: ['windows'],
});

const MAX_RETRIES = 3;
const RETRY_DELAY = 5000;

const checkAdidas = async (urlConfig) => {
  const { name, url, sizes } = urlConfig;
  const fingerprint = fingerprintGenerator.getFingerprint({
    locales: ['en-US', 'en'],
    screen: { width: 1920, height: 1080 },
  });

  let browser;
  let retryCount = 0;

  while (retryCount < MAX_RETRIES) {
    try {
      console.log(`Pokus ${retryCount + 1}/${MAX_RETRIES} pre URL: ${url}`);
      browser = await puppeteer.launch({
        headless: 'new',
        args: [
          `--proxy-server=${proxyWebShare.host}:${proxyWebShare.port}`,
          "--no-sandbox",
          "--disable-setuid-sandbox",
        ],
        ignoreHTTPSErrors: true,
      });

      const [page] = await browser.pages();
      await page.authenticate({
        username: proxyWebShare.user,
        password: proxyWebShare.pass,
      });

      await page.setUserAgent(fingerprint.fingerprint.navigator.userAgent);
      await page.setViewport({
        width: fingerprint.fingerprint.screen.width,
        height: fingerprint.fingerprint.screen.height,
        deviceScaleFactor: fingerprint.fingerprint.screen.deviceScaleFactor ?? 1,
      });

      await page.setExtraHTTPHeaders({
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'Connection': 'keep-alive',
        'Upgrade-Insecure-Requests': '1'
      });

      await page.goto(url, {
        waitUntil: ['networkidle0', 'domcontentloaded'],
        timeout: 80000,
        referer: 'https://www.adidas.cz/',
      });

      // 后续的页面处理逻辑...

    } catch (error) {
      console.error(`Pokus ${retryCount + 1} selhal:`, error);
      retryCount++;
      if (browser) await browser.close();
      if (retryCount < MAX_RETRIES) {
        console.log(`Znovu zkusíme za ${RETRY_DELAY / 1000} sekund...`);
        await new Promise(resolve => setTimeout(resolve, RETRY_DELAY));
      }
    }
  }

  if (browser) await browser.close();
};

想请教大家：

1. 我的代码里有没有遗漏的反检测配置？
2. 针对Adidas的反爬，还有哪些无头模式下的特殊处理手段？
3. 有没有可能是代理或者指纹的配置有问题？

非常感谢大家的帮助！

内容来源于stack exchange