使用Puppeteer.connect搭配代理时如何设置User Agent？

使用puppeteer.connect连接browserless.io远程浏览器服务，通过page.authenticate完成远程代理认证，但每次尝试设置User Agent均失败。移除代理后所有功能正常运作，请问该场景下如何设置User Agent？

相关代码示例 #

async function taskProcessor(task) {
  const browser = await puppeteer.connect({ browserWSEndpoint: 'wss://chrome.browserless.io/?token=b&stealth&headless=false&slowMo=1000&ignoreDefaultArgs=--hide-scrollbars,--disable-default-apps&--proxy-server=https://br-pr.oxylabs.io:20000'})
  const page = await browser.newPage();
    await page.authenticate({
      username: 'user',
      password: 'password'
    })

  const userAgent = new UserAgent();

  console.log(userAgent.toString());
  
  await page.setViewport({width:1920,height:1080});

  await page.setUserAgent(userAgent.toString());

  await page.goto('https://google.com',{waitUntil:"networkidle2"});

  const userAgentString = await page.evaluate(() => navigator.userAgent);
  console.log(userAgentString);
}

输出结果 #

2023-07-25T23:38:31.139741+00:00 app[web.1]: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36
2023-07-25T23:38:45.331961+00:00 app[web.1]: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36

问题核心是browserless的stealth参数会覆盖自定义User Agent，同时代理服务可能篡改请求头，可通过以下方式解决：

方法一：关闭stealth并通过启动参数指定User Agent #

移除browserless连接URL中的stealth参数，直接在启动参数里添加--user-agent，确保浏览器启动时就应用自定义User Agent：

async function taskProcessor(task) {
  const customUserAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36';
  const browser = await puppeteer.connect({ 
    browserWSEndpoint: `wss://chrome.browserless.io/?token=b&headless=false&slowMo=1000&ignoreDefaultArgs=--hide-scrollbars,--disable-default-apps&--proxy-server=https://br-pr.oxylabs.io:20000&--user-agent=${encodeURIComponent(customUserAgent)}`
  });
  const page = await browser.newPage();
  await page.authenticate({
    username: 'user',
    password: 'password'
  });

  await page.setViewport({width:1920,height:1080});

  await page.goto('https://google.com',{waitUntil:"networkidle2"});

  const userAgentString = await page.evaluate(() => navigator.userAgent);
  console.log(userAgentString);
}

方法二：保留stealth，通过请求头和页面注入修改 #

如果需要stealth功能，同时设置HTTP请求头的User Agent，并在页面初始化时修改navigator.userAgent的只读属性：

async function taskProcessor(task) {
  const customUserAgent = new UserAgent().toString();
  console.log(customUserAgent);

  const browser = await puppeteer.connect({ 
    browserWSEndpoint: 'wss://chrome.browserless.io/?token=b&stealth&headless=false&slowMo=1000&ignoreDefaultArgs=--hide-scrollbars,--disable-default-apps&--proxy-server=https://br-pr.oxylabs.io:20000'
  });
  const page = await browser.newPage();
  
  await page.authenticate({
    username: 'user',
    password: 'password'
  });

  await page.setViewport({width:1920,height:1080});

  // 设置HTTP请求头中的User Agent
  await page.setExtraHTTPHeaders({
    'User-Agent': customUserAgent
  });

  // 在页面加载前注入代码，修改navigator.userAgent
  await page.evaluateOnNewDocument((ua) => {
    Object.defineProperty(navigator, 'userAgent', {
      get: () => ua,
      configurable: true
    });
  }, customUserAgent);

  await page.goto('https://google.com',{waitUntil:"networkidle2"});

  const userAgentString = await page.evaluate(() => navigator.userAgent);
  console.log(userAgentString);
}

方法三：检查代理服务设置 #

部分代理服务商（如Oxylabs）可能默认篡改User Agent，可登录代理后台查看并关闭相关设置。

内容的提问来源于stack exchange，提问作者Pedro Fontes