Puppeteer@24.1.0+/Chrome@132.0+（headless:true模式）访问Basic Auth页面时出现ERR_BLOCKED_BY_CLIENT错误求助

各位好，我最近在测试爬取客户服务站点的时候遇到了一个奇怪的问题：当使用Puppeteer@24.1.0搭配Chrome@132.0.6834.83，并且开启headless: true模式时，无法访问带Basic Auth认证的页面，会抛出net::ERR_BLOCKED_BY_CLIENT错误。我已经在自己的开发环境里用最小测试用例复现了这个问题，想请教大家：这是不是新版本Puppeteer/Chrome需要额外的配置，还是软件出现了退化，或者有其他我没考虑到的原因？

测试环境 #

• OS: Rocky Linux 9.5
• Web服务器: Apache httpd 2.4.62
• Node.js: 20.18.2（来自nodesource）

Puppeteer/Chrome版本测试情况 #

正常可用版本列表 #

这些版本都能正常通过用户名/密码访问带Basic Auth的页面：

• puppeteer@22.15.0（headless: true）
• puppeteer@23.11.1/chrome@131.0.6778.204（headless: true）
• puppeteer@23.11.1/chrome-headless-shell@131.0.6778.204（headless: 'shell'）
• puppeteer@24.0.0/chrome@131.0.6778.264（headless: true）
• puppeteer@24.0.0/chrome-headless-shell@131.0.6778.264（headless: 'shell'）
• puppeteer@24.1.0/chrome-headless-shell@132.0.6834.83（headless: 'shell'）
• puppeteer@24.2.0/chrome-headless-shell@133.0.6943.53（headless: 'shell'）

报错版本列表 #

以下版本访问带Basic Auth的页面会失败：

• puppeteer@24.1.0/chrome@132.0.6834.83（headless: true）
• puppeteer@24.2.0/chrome@133.0.6943.53（headless: true）

错误堆栈信息 #

net::ERR_BLOCKED_BY_CLIENT  
at http://[myhost]/[mydir]/ 
at navigate (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Frame.js:183:27) 
at async Deferred.race (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/util/Deferred.js:36:20) 
at async CdpFrame.goto (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Frame.js:149:25) 
at async CdpPage.goto (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/api/Page.js:574:20) 
at async [my_js]

完整复现步骤与代码 #

Apache Basic Auth配置（Rocky Linux 9.5） #

创建配置文件/etc/httpd/conf.d/basic_auth.conf：

<Directory "/var/www/html/basic_auth">
AuthType Basic
AuthName "Restricted"
AuthUserFile "/mypwdir/htpasswd"
Require valid-user
</Directory>

生成密码文件的命令：

# htpasswd -c /mypwdir/htpasswd testuser
# 提示时输入测试密码即可

测试页面说明 #

页面内容不影响测试结果，即使是空文件（大小0）也会触发报错。示例页面/var/www/html/basic_auth/test.html：

<html>
<head><title>test page</title></head>
<body>test page</body>
</html>

Puppeteer测试脚本（pt_test1.js） #

// reproducer: puppeteer >= 24.1.1 ERR_BLOCKED_BY_CLIENT w/Basic auth

'use strict';

const ptdelay = 0;
const pttmo = 10000; // msec.

const puppeteer = require('puppeteer');
const fs = require('fs');

process.on('unhandledRejection', (reason, p) => {
  console.log('Error:unhandled Rejection at: Promise '+p+' reason: '+reason);
  process.exit();
});

// /usr/bin/node = process.argv[0];
const jsfile = process.argv[1];
var url      = process.argv[2]; // url
if (url === undefined) { ERX('no url'); }
var uname    = process.argv[3]; // -wwwauth username:password
var passwd   = process.argv[4]; // -wwwauth username:password

// for basic auth
var wwwauth  = {};
if (uname)  { wwwauth.username = uname; }
if (passwd) { wwwauth.password = passwd; }

// chrome arguments
var ptargs = [];
ptargs.push('--no-sandbox');
ptargs.push('--disable-setuid-sandbox');
//ptargs.push('--ignore-certificate-errors'); //*** problem disappears with this option ***//

// puppeteer options
var ptopts = {
 args: ptargs
,headless: true
,env: {}
//,ignoreHTTPSErrors: false
};

console.log('-');
console.log('url:['+url+']');
console.log('-');

// main
( async () => {
  const browser = await puppeteer.launch(ptopts);
  const page = await browser.newPage();

  // basic auth
  if (Object.keys(wwwauth).length > 0) {
    await page.authenticate(wwwauth);
  }

  var tstart = Date.now();
  var response;
  try {
    response = await page.goto(url, {timeout: pttmo, waituntil: 'load'});
  } catch (e) {
    console.log(e);
    browser.close();
    process.exit();
  }
  await console.log('HTTP/1.1 '+response.status()+' '+'FILLER');
  await console.log('-');

  var h = response.headers();
  Object.keys(h).forEach(function(key){
    console.log(key+':'+this[key]);
  }, h);
  await sleep(ptdelay);
  
  const c = await page.content();
  await console.log('-');
  await console.log(c);
  await console.log("\n");
  await console.log('-');
  await console.log((Date.now() - tstart)/1000);

  await page.close();
  await browser.close();
})();

function ERX(emsg) {
  console.log('ERROR: '+emsg);
  console.log('Usage: node '+jsfile+' <url> [<username> <password>]');
  process.exit();
}

function sleep(time) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      resolve();
    }, time);
  });
}

运行示例1（报错情况） #

# /usr/bin/node /mydir/pt_test1.js http://mydomain/basic_auth/test.html testuser testpw
-
url:[http://mydomain/basic_auth/test.html]
-
Error: net::ERR_BLOCKED_BY_CLIENT at http://mydomain/basic_auth/test.html
    at navigate (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Frame.js:183:27)
    at async Deferred.race (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/util/Deferred.js:36:20)
    at async CdpFrame.goto (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Frame.js:149:25)
    at async CdpPage.goto (/mylib/node_modules/puppeteer-core/lib/cjs/puppeteer/api/Page.js:574:20)
    at async /mydir/pt_test1.js:63:16

运行示例2（添加--ignore-certificate-errors后恢复正常） #

取消脚本中--ignore-certificate-errors的注释后运行，问题消失：

# /usr/bin/node /mydir/pt_test1.js http://mydomain/basic_auth/test.html testuser testpw
-
url:[http://mydomain/basic_auth/test.html]
-
HTTP/1.1 200 FILLER
-
accept-ranges:bytes
connection:Keep-Alive
content-length:0
content-type:text/html; charset=UTF-8
date:Thu, 06 Feb 2025 06:30:43 GMT
etag:"0-62d73543364b9"
keep-alive:timeout=5, max=99
last-modified:Thu, 06 Feb 2025 06:26:57 GMT
server:Apache/2.4.62 (Rocky Linux) OpenSSL/3.0.7 mod_fcgid/2.3.9 mod_perl/2.0.12 Perl/v5.32.1
strict-transport-security:max-age=31536000; includeSubDomains; preload
-
<html><head></head><body></body></html>


-
0.137

备注：内容来源于stack exchange，提问作者BSTn6