Ubuntu服务器中Puppeteer访问TikTok直播被检测，本地正常求解决

问题描述 #

本地运行Puppeteer脚本观看TikTok直播一切正常，但部署到Ubuntu 20.04 LTS Server后，直播页面能加载却无法启动，还要求登录（本地无此要求）。即使给服务器安装GUI手动访问直播仍无法观看，推测是服务器环境被TikTok的反爬机制识别。

原脚本配置：

const puppeteer = require("puppeteer-extra");
const { Cluster } = require("puppeteer-cluster");

// Use stealth plugin to bypass bot detection
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
const AnonymizeUA = require("puppeteer-extra-plugin-anonymize-ua");
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');

puppeteer.use(StealthPlugin());
puppeteer.use(AnonymizeUA());
puppeteer.use(AdblockerPlugin({ blockTrackers: true }));


(async () => {
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT,
    maxConcurrency: 60000,
    timeout: 86400000,
    puppeteer: puppeteer,
    retryLimit: 10,
    retryDelay: 1000,
    puppeteerOptions: {
      headless: true,
      timeout: 120000, //360000
      args: [
        "--start-maximized",
        "--no-sandbox",
        "--disable-setuid-sandbox",
        "--disable-dev-shm-usage",
        "--disable-accelerated-2d-canvas",
        "--no-first-run",
        "--no-zygote",
        "--disable-gpu",
      ],
      executablePath: "/snap/bin/chromium",
      defaultViewport: null,
    },
  });

绕过检测的解决方案 #

1. 优化浏览器指纹模拟 #

• 移除AnonymizeUA插件，手动设置真实桌面Chrome的用户代理字符串，避免随机UA被识别：

  // 替换原AnonymizeUA配置，添加到puppeteerOptions.args中
  '--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"'
  

• 配置StealthPlugin启用全量反检测规则，覆盖更多指纹检测点：

  puppeteer.use(StealthPlugin({
    enabledEvasions: [
      'chrome.runtime',
      'iframe.contentWindow',
      'media.codecs',
      'navigator.hardwareConcurrency',
      'navigator.languages',
      'navigator.permissions',
      'navigator.plugins',
      'navigator.vendor',
      'navigator.webdriver',
      'sourceurl',
      'user-agent-override',
      'webgl.vendor',
      'window.outerdimensions'
    ]
  }));
  

2. 调整Chromium启动参数 #

添加模拟真实桌面环境的参数，移除暴露服务器特征的配置：

args: [
  "--no-sandbox",
  "--disable-setuid-sandbox",
  "--disable-dev-shm-usage",
  "--window-size=1920,1080", // 替换--start-maximized，贴近真实窗口尺寸
  "--lang=zh-CN,zh", // 设置真实语言环境
  "--enable-features=NetworkService,NetworkServiceInProcess",
  "--disable-features=site-per-process,TranslateUI,BlinkGenPropertyTrees",
  "--disable-webrtc", // 禁用WebRTC防止暴露真实IP
  "--ignore-certificate-errors",
  "--disable-blink-features=AutomationControlled", // 阻止Chrome暴露自动化状态
  "--enable-blink-features=IdleDetection", // 模拟用户活跃状态
]

注意：移除--no-zygote，该参数会让Chromium运行模式偏离常规桌面环境，易被检测。

3. 降低集群并发数 #

原maxConcurrency: 60000远超服务器承载能力，极端高并发会直接触发流量检测。根据服务器配置设置合理值（比如2-4）：

maxConcurrency: 2,

4. 使用真实用户数据目录 #

将本地Chrome的用户数据目录（Windows：C:\Users\<用户名>\AppData\Local\Google\Chrome\User Data；macOS：~/Library/Application Support/Google/Chrome）复制到服务器，在脚本中指定该目录，让浏览器拥有真实缓存、Cookie和浏览历史：

puppeteerOptions: {
  // ...其他配置
  userDataDir: "/path/to/your/local-chrome-user-data",
}

5. 模拟真实用户交互 #

页面加载后添加模拟操作，避免被识别为无交互爬虫：

// 在Cluster任务处理函数中添加：
await page.waitForSelector('button[data-e2e="live-play-button"]', { timeout: 10000 });
await page.click('button[data-e2e="live-play-button"]');
// 模拟滚动
await page.evaluate(() => window.scrollBy(0, 200));
// 等待直播加载
await page.waitForTimeout(5000);

6. 网络IP伪装 #

使用住宅代理替换服务器IP，避免服务器IP被标记为非用户IP段，在启动参数中添加：

args: [
  // ...其他参数
  "--proxy-server=http://your-residential-proxy:port",
]

修改后的完整脚本片段 #

const puppeteer = require("puppeteer-extra");
const { Cluster } = require("puppeteer-cluster");

const StealthPlugin = require("puppeteer-extra-plugin-stealth");
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');

// 配置Stealth插件启用全量反检测规则
puppeteer.use(StealthPlugin({
  enabledEvasions: [
    'chrome.runtime',
    'iframe.contentWindow',
    'media.codecs',
    'navigator.hardwareConcurrency',
    'navigator.languages',
    'navigator.permissions',
    'navigator.plugins',
    'navigator.vendor',
    'navigator.webdriver',
    'sourceurl',
    'user-agent-override',
    'webgl.vendor',
    'window.outerdimensions'
  ]
}));
puppeteer.use(AdblockerPlugin({ blockTrackers: true }));

(async () => {
  const cluster = await Cluster.launch({
    concurrency: Cluster.CONCURRENCY_CONTEXT,
    maxConcurrency: 2, // 合理设置并发数
    timeout: 86400000,
    puppeteer: puppeteer,
    retryLimit: 3, // 降低重试次数，避免频繁请求触发检测
    retryDelay: 3000,
    puppeteerOptions: {
      headless: "new", // 使用新版无头模式，更贴近真实浏览器
      timeout: 120000,
      args: [
        "--no-sandbox",
        "--disable-setuid-sandbox",
        "--disable-dev-shm-usage",
        "--window-size=1920,1080",
        "--lang=zh-CN,zh",
        "--enable-features=NetworkService,NetworkServiceInProcess",
        "--disable-features=site-per-process,TranslateUI,BlinkGenPropertyTrees",
        "--disable-webrtc",
        "--ignore-certificate-errors",
        "--disable-blink-features=AutomationControlled",
        "--enable-blink-features=IdleDetection",
        '--user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"'
      ],
      executablePath: "/snap/bin/chromium",
      defaultViewport: null,
      userDataDir: "/path/to/imported-chrome-user-data" // 替换为实际用户数据目录路径
    },
  });

  // 添加任务处理逻辑
  await cluster.task(async ({ page, data: liveUrl }) => {
    await page.goto(liveUrl, { waitUntil: "networkidle2" });
    // 模拟用户交互
    await page.waitForSelector('button[data-e2e="live-play-button"]', { timeout: 10000 });
    await page.click('button[data-e2e="live-play-button"]');
    await page.evaluate(() => window.scrollBy(0, 200));
    await page.waitForTimeout(5000);
    // 后续逻辑...
  });

  // 传入直播URL任务
  await cluster.queue("https://www.tiktok.com/@username/live");
})();

内容的提问来源于stack exchange，提问作者Mike