如何阻止Selenium Chrome(Remote WebDriver)+NetworkInterceptor升级HTTP为HTTPS

问题背景 #

使用Selenium Hub搭配node-chromium实例，通过NetworkInterceptor添加自定义请求头时，所有HTTP站点请求会被自动升级为HTTPS，最终返回空白页面；但不使用NetworkInterceptor时请求可正常获取HTTP站点内容，说明问题与请求头拦截逻辑直接相关。

环境配置（Docker Compose） #

version: '3.8'
name: 'selenium-feature-extractor-dev'
services:
  selenium-hub:
    image: seleniarm/hub:4.18.1-20240327
    ports:
      - "4444:4444"
  selenium-chrome:
    image: seleniarm/node-chromium:123.0-chromedriver-123.0-grid-4.18.1-20240327
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=10
      - SE_OPTS=--grid-url http://localhost:4444

请求头拦截代码 #

try (NetworkInterceptor ignored = addHeadersToRequests(driver, headers)) {
   ...
}

private NetworkInterceptor addHeadersToRequests(WebDriver driver, Map<String, String> headers) {
    if (driver instanceof HasDevTools && headers != null) {
        Filter filter = next -> req -> {
            headers.forEach((key, value) -> {
                req.removeHeader(key);
                req.addHeader(key, value);
            });
            return next.execute(req);
        };
        return new NetworkInterceptor(driver, filter);
    }
    return null;
}

关键日志信息 #

日志中可见请求头包含Upgrade-Insecure-Requests: 1，这是触发HTTP自动升级到HTTPS的核心原因：

2024-04-03T15:51:27.558+01:00  INFO 26651 --- [ CDP Connection] o.openqa.selenium.devtools.Connection    : <- {"method":"Fetch.requestPaused","params":{"requestId":"interception-job-1.0","request":{"url":"https://mysite.pro/","method":"GET","headers":{"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","Accept-Language":"en-GB,en-US;q=0.9,en;q=0.8","Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"},"initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin"},"frameId":"D3EC8B769B07AD51B5752B137EF1916B","resourceType":"Document"},"sessionId":"B0A124EC0331A37418F1136BB92D1F32"}

已配置的Chrome启动参数 #

"--disable-dev-shm-usage",
"--ignore-certificate-errors",
"--disable-blink-features=AutomationControlled",
"--log-level=DEBUG",
"--whitelisted-ips",
"--disable-gpu",
"--disable-software-rasterizer",
"--verbose",
"--remote-allow-origins=*",
"--ignore-urlfetcher-cert-requests",
"--allow-running-insecure-content"

解决方法 #

1. 在拦截逻辑中移除升级触发头 #

修改NetworkInterceptor的filter逻辑，强制移除Upgrade-Insecure-Requests请求头，直接阻止浏览器触发HTTP升级：

private NetworkInterceptor addHeadersToRequests(WebDriver driver, Map<String, String> headers) {
    if (driver instanceof HasDevTools && headers != null) {
        Filter filter = next -> req -> {
            headers.forEach((key, value) -> {
                req.removeHeader(key);
                req.addHeader(key, value);
            });
            // 移除触发HTTP升级的请求头
            req.removeHeader("Upgrade-Insecure-Requests");
            return next.execute(req);
        };
        return new NetworkInterceptor(driver, filter);
    }
    return null;
}

2. 添加Chrome参数彻底禁用自动升级 #

在已有的ChromeOptions启动参数中，补充以下参数，从浏览器层面禁用自动升级行为：

"--disable-features=AutoupgradeMixedContent",
"--disable-upgrade-insecure-requests"

3. 验证请求URL的正确性 #

确认业务代码中请求的是明确的http://开头的URL，排除代码逻辑误将HTTP地址转为HTTPS的情况。

内容的提问来源于stack exchange，提问作者ndtreviv