Java Htmlunit 2.20 绕过Cloudflare验证卡滞问题求助

问题概述 #

• 受公司政策限制必须使用Htmlunit 2.20版本，访问巴西政府ANEEL网站：https://www2.aneel.gov.br:443/aplicacoes_liferay/tarifa/
• 普通浏览器访问正常，但Htmlunit触发Cloudflare的"Checking if the site connection is secure"验证流程，卡在"Connection is secure - Proceeding..."步骤无法跳转至目标页面
• 无法将IP加入白名单，必须自行完成验证流程
• 调整BrowserVersion、Cookie等配置后问题仍未解决，还出现cf_chl_2 Cookie的max-age负数错误：

[com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl] (JS executor for com.gargoylesoftware.htmlunit.WebClient@724c8784) set-cookie http-equiv meta tag: invalid cookie 'cf_chl_2=; Max-Age=-99999999;'; reason: 'Negative 'max-age' attribute: -99999999'.

现有代码示例：

BrowserVersion chrome = new BrowserVersion(
            "Netscape", 
            "5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36", 
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36", 
            108);
    chrome.setApplicationCodeName("Mozilla");
    chrome.setVendor("Google Inc.");
    chrome.setHtmlAcceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
    chrome.setImgAcceptHeader("image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8");
    chrome.setCssAcceptHeader("text/css,*/*;q=0.1");
    chrome.setScriptAcceptHeader("*/*");
    chrome.setBrowserLanguage("en-US,en;q=0.9,pt;q=0.8,mt;q=0.7");
    chrome.setPlatform("Windows");
    chrome.setUserLanguage("pt-BR");
    chrome.setSystemLanguage("pt-BR");
    
    try (WebClient webClient = new WebClient(chrome)) {
        String url = "https://www2.aneel.gov.br:443/aplicacoes_liferay/tarifa/";
        
        // parâmetros do webclient
        webClient.getOptions().setCssEnabled(true);
        webClient.setJavaScriptTimeout(0);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setUseInsecureSSL(false);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
        webClient.getOptions().setTimeout(0);
        
        CookieManager cookies = new CookieManager();
        
        cookies.setCookiesEnabled(true);
        webClient.setCookieManager(cookies);
        
        
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setRedirectEnabled(true);
        
        webClient.waitForBackgroundJavaScript(10000);
        webClient.waitForBackgroundJavaScriptStartingBefore(10000);
        
        webClient.getCache().setMaxSize(0);
        
        HtmlPage page = webClient.getPage(url);
        webClient.getRefreshHandler().handleRefresh(page, new URL(url), 10);
        
        synchronized(page) {
            page.wait(10000);
        }
        
        URL _url = new URL(url);
        
        for(Cookie c : webClient.getCookies(_url)) {
            if (c.getName().contains("cf_chl_2")) {
                
                Cookie cook = new Cookie(c.getDomain(), c.getName(), c.getValue(), c.getPath(), -1, false);
                webClient.getCookieManager().removeCookie(c);
                webClient.getCookieManager().addCookie(cook);
            }
        }
        
        if (page.asXml().contains("Checking if the site connection is secure")) {
            log.info(page.asXml());
            
            page = webClient.getPage(url);
            webClient.waitForBackgroundJavaScript(10_000);
        }

可行解决方法 #

1. 忽略负max-age的Cookie错误 #

Cloudflare返回的负max-agecf_chl_2 Cookie是标准的清除旧Cookie的方式，Htmlunit 2.20对这类Cookie的校验过于严格，可通过自定义监听器忽略该错误：

webClient.setIncorrectnessListener(new IncorrectnessListener() {
    @Override
    public void notify(String message, Object origin) {
        // 仅忽略cf_chl_2的负max-age错误，其他错误正常输出
        if (!message.contains("Negative 'max-age' attribute") || !message.contains("cf_chl_2")) {
            System.err.println(message);
        }
    }
});

注意：不要手动修改cf_chl_2 Cookie，会干扰Cloudflare的验证流程。

2. 优化JS等待逻辑，确保验证脚本执行完成 #

Htmlunit 2.20的synchronized(page).wait()不会等待后台JS执行，需改用waitForBackgroundJavaScript并延长等待时间，确保Cloudflare的验证脚本完全运行：

HtmlPage page = webClient.getPage(url);
// 等待Cloudflare验证JS执行，延长至20秒
webClient.waitForBackgroundJavaScript(20000);

// 若仍在验证页面，等待几秒后重新请求
if (page.asXml().contains("Checking if the site connection is secure")) {
    Thread.sleep(5000);
    page = webClient.getPage(url);
    webClient.waitForBackgroundJavaScript(20000);
}

同时移除手动调用handleRefresh，让WebClient自动处理跳转。

3. 完善BrowserVersion配置，贴近真实浏览器 #

调整请求头和语言设置，更符合巴西地区真实Chrome浏览器的特征：

// 修改浏览器语言为巴西葡萄牙语优先
chrome.setBrowserLanguage("pt-BR,pt;q=0.9,en;q=0.8");
// 添加真实浏览器的额外请求头
chrome.setAdditionalHeader("Accept-Encoding", "gzip, deflate, br");
chrome.setAdditionalHeader("Connection", "keep-alive");
chrome.setAdditionalHeader("DNT", "1");

4. 简化Cookie和缓存配置 #

保持缓存禁用，但不要手动操作CookieManager，让WebClient自动管理Cloudflare的验证Cookie。

修改后的完整代码示例 #

BrowserVersion chrome = new BrowserVersion(
        "Netscape", 
        "5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36", 
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36", 
        108);
chrome.setApplicationCodeName("Mozilla");
chrome.setVendor("Google Inc.");
chrome.setHtmlAcceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
chrome.setImgAcceptHeader("image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8");
chrome.setCssAcceptHeader("text/css,*/*;q=0.1");
chrome.setScriptAcceptHeader("*/*");
chrome.setBrowserLanguage("pt-BR,pt;q=0.9,en;q=0.8");
chrome.setPlatform("Windows");
chrome.setUserLanguage("pt-BR");
chrome.setSystemLanguage("pt-BR");
// 添加真实浏览器请求头
chrome.setAdditionalHeader("Accept-Encoding", "gzip, deflate, br");
chrome.setAdditionalHeader("Connection", "keep-alive");
chrome.setAdditionalHeader("DNT", "1");

try (WebClient webClient = new WebClient(chrome)) {
    String url = "https://www2.aneel.gov.br:443/aplicacoes_liferay/tarifa/";
    
    // WebClient核心配置
    webClient.getOptions().setCssEnabled(true);
    webClient.setJavaScriptTimeout(30000);
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    webClient.getOptions().setUseInsecureSSL(false);
    webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    webClient.getOptions().setTimeout(30000);
    webClient.getOptions().setJavaScriptEnabled(true);
    webClient.getOptions().setRedirectEnabled(true);
    
    // 忽略cf_chl_2的负max-age错误
    webClient.setIncorrectnessListener(new IncorrectnessListener() {
        @Override
        public void notify(String message, Object origin) {
            if (!message.contains("Negative 'max-age' attribute") || !message.contains("cf_chl_2")) {
                System.err.println("Htmlunit Incorrectness: " + message);
            }
        }
    });
    
    // 禁用缓存
    webClient.getCache().setMaxSize(0);
    
    // 请求页面并等待验证完成
    HtmlPage page = webClient.getPage(url);
    webClient.waitForBackgroundJavaScript(20000);
    
    // 二次验证跳转
    if (page.asXml().contains("Checking if the site connection is secure")) {
        Thread.sleep(5000);
        page = webClient.getPage(url);
        webClient.waitForBackgroundJavaScript(20000);
    }
    
    // 输出目标页面内容
    System.out.println(page.asText());
} catch (Exception e) {
    e.printStackTrace();
}

内容的提问来源于stack exchange，提问作者gbossa