如何使用HTMLUnit捕获页面渲染时触发的所有请求？

问题 #

使用HTMLUnit加载本地HTML文件时，希望捕获所有触发的请求。Chrome渲染该文件时Network标签显示共7个请求，但当前代码仅输出1个请求就结束。

测试HTML文件：

<script type="text/javascript">
  !(function () {
    var adc = function (str) {
      return decodeURIComponent(escape(window.atob(str)));
    };
    document.write(
      adc(
        "PGEgaHJlZj0iaHR0cHM6Ly9kc3A4dTRqaGE0NHJyLmNsb3VkZnJvbnQubmV0L2RpcmVjdC8xOTgyMzQxMDE/YWR4PUFsZ29yaXgoUHJvKSZhcHA9MjgxOTQwMjkyJnByaWNlPTAuOTEwMSZyZD1MVldrTUJRTUxGQzhrIiB0YXJnZXQ9Il9ibGFuayI+PGltZyBzcmM9Imh0dHBzOi8vZHNwOHU0amhhNDRyci5jbG91ZGZyb250Lm5ldC9pbXAvMTk4MjM0MTAxP2FkeD1BbGdvcml4KFBybykmYXBwPTI4MTk0MDI5MiZwcmljZT0wLjkxMDEmcmQ9TFZXa01CUU1MRkM4ayIgd2lkdGg9IjMyMCIgaGVpZ2h0PSI1MCI+PC9hPjxpbWcgc3JjPSJodHRwczovL2QybWsybmg4dmZmNzY4LmNsb3VkZnJvbnQubmV0L3YxL3BpeGVsP2E9MTAyOSZiPTEwNDYmYz0xJmQ9ZmZkNDkyNTFjYjkwOGI5NSZlPTU4Njg0YWYzN2Q0MTFhNGImZj0wLjkxMDEmZz0wLjkxMDE0Jmg9MTA0NSZpPWZiMTJmZjY5ZjJkZmFiZjAmaz04MzM1NTEzMTAxMDI5ODI2NjAmcmQ9TFZXa01CUU1MRkM4ayIgYm9yZGVyPSIwIiB3aWR0aD0iMSIgaGVpZ2h0PSIxIi8+"
      )
    );
    document.write(
      adc(
        "PGltZyBzcmM9Imh0dHBzOi8vdXNlLnRyay5zdnItYWxnb3JpeC5jb20vaW1wP2NycHY9MyZpbmZvPTlFbVpwWkNNdWdETnVFak14NHlNeTBEY3BWbkp5a2pNd1FUT3hnak05UVdkaVpDTTlRSGR5Tm5KeDBEZDBsbVltRVRQdEJuWW1Bek45STNjeU5uSngwVGJtQm5KdzBEYzRWbUp3MFRhd0ZtSngwVGUwRm1KelFUTndZVFBrbDJjbWdUTTVJak41RURPMkVUUDBKbkp4a2pMdzBUYmhaU000TWpOdUFUUHRaU013RVRPdUFUUHRKbUp3a3pNOWtuWW1FMFVWMXpZbUV6TTJRek54MERjbVEyTmhKV04xUWpNelUyTTNnell3Z1RZalZHTndVR09rSlRZNVVUTW1oVFo5RW5jJnByaWNlPSR7QVVDVElPTl9QUklDRX0mcz02MDU0MyZyPWU4ZjE1OWEyZDhlMDRlY2E4MGM4NzNlMzI0NTViYTdkIiB3aWR0aD0iMSIgaGVpZ2h0PSIxIiBzdHlsZT0iZGlzcGxheTpub25lOyI+PGRpdiBpZD0iZG9qczIwMTJiMDVhIiBkYXRhLXdpZHRoPSIzMjAiIGRhdGEtaGVpZ2h0PSI1MCIgZGF0YS10cms9J2h0dHBzOi8vdXNlLnRyay5zdnItYWxnb3JpeC5jb20vaW1wP2NycHY9MyZpbmZvPTlFbVpwWkNNdWdETnVFak14NHlNeTBEY3BWbkp5a2pNd1FUT3hnak05UVdkaVpDTTlRSGR5Tm5KeDBEZDBsbVltRVRQdEJuWW1Bek45STNjeU5uSngwVGJtQm5KdzBEYzRWbUp3MFRhd0ZtSngwVGUwRm1KelFUTndZVFBrbDJjbWdUTTVJak41RURPMkVUUDBKbkp4a2pMdzBUYmhaU000TWpOdUFUUHRaU013RVRPdUFUUHRKbUp3a3pNOWtuWW1FMFVWMXpZbUV6TTJRek54MERjbVEyTmhKV04xUWpNelUyTTNnell3Z1RZalZHTndVR09rSlRZNVVUTW1oVFo5RW5jJnByaWNlPSR7QVVDVElPTl9QUklDRX0mcz02MDU0MyZyPWU4ZjE1OWEyZDhlMDRlY2E4MGM4NzNlMzI0NTViYTdkJyBkYXRhLWlkPSdBbGdvcmlYLWU4ZjE1OWEyZDhlMDRlY2E4MGM4NzNlMzI0NTViYTdkJz48c2NyaXB0IHR5cGU9J3RleHQvamF2YXNjcmlwdCcgYXN5bmMgc3JjPSJodHRwczovL3Ryay5zdnItYWxnb3JpeC5jb20vc3RhdGljL200LmpzP3Q9OTM0NDIzIj48L3NjcmlwdD48L2Rpdj4="
      ).replace(new RegExp(adc("XCR7QVVDVElPTl9QUklDRX0="), "g"), "0.6381")
    );
  })();
</script>
<img
  src="https://use.trk.svr-algorix.com/win?crpv=3&info=9EmZpZCMugDNuEjMx4yMy0DcpVnJykjMwQTOxgjM9QWdiZCM9QHdyNnJx0Dd0lmYmETPtBnYmAzN9I3cyNnJx0TbmBnJw0Dc4VmJw0TawFmJx0Te0FmJzQTNwYTPkl2cmgTM5IjN5EDO2ETP0JnJxkjLw0TbhZSM4MjNuATPtZSMwETOuATPtJmJwkzM9knYmE0UV1zYmEzM2QzNx0DcmQ2NhJWN1QjMzU2M3gzYwgTYjVGNwUGOkJTY5UTMmhTZ9Enc&price=0.6381&s=60543&r=e8f159a2d8e04eca80c873e32455ba7d"
  width="1"
  height="1"
  style="display: none"
/>

当前Java代码：

public class RenderHTML extends WebConnectionWrapper {

    static List<String> list = new ArrayList<String>();


    public RenderHTML(WebClient webClient) throws IllegalArgumentException {
        super(webClient);
    }

    @Override
    public WebResponse getResponse(WebRequest request) throws IOException {
        // Log the URL of the request
        System.out.println(request.getUrl().toString());
        return super.getResponse(request);
    }



    public static void main(String[] args) throws IOException {
        try (WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
            // Wrap the client with the URLRecorder
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.waitForBackgroundJavaScriptStartingBefore(100_000);
            webClient.waitForBackgroundJavaScript(100_000);
            webClient.getOptions().setCssEnabled(true);
            webClient.getOptions().setRedirectEnabled(true);
            webClient.getOptions().setUseInsecureSSL(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.getCookieManager().setCookiesEnabled(true);
            webClient.setAjaxController(new AjaxController());
            webClient.getCookieManager().setCookiesEnabled(true);

            webClient.setWebConnection(new RenderHTML( webClient));

            // Load the local HTML file
            HtmlPage page = webClient.getPage("file:///Users/derrickguo/work/project/project_java/analyze_demand_tool_maven/src/main/lib/algorix_us_adm.html");
            
        }
    }
}

解决方案 #

问题核心是JavaScript执行时机未正确等待，导致动态生成的请求还未触发就结束了程序。调整代码如下：

1. 调整等待JavaScript的位置 #

waitForBackgroundJavaScript和waitForBackgroundJavaScriptStartingBefore需要放在页面加载完成之后调用，而不是之前。这些方法会等待后台JavaScript任务执行完毕，确保所有动态请求都被触发。

2. 补充等待逻辑 #

添加webClient.waitForJavaScriptJobs()确保所有异步JavaScript任务完成，包括document.write插入内容后触发的资源加载。

修改后的完整代码 #

public class RenderHTML extends WebConnectionWrapper {

    public RenderHTML(WebClient webClient) throws IllegalArgumentException {
        super(webClient);
    }

    @Override
    public WebResponse getResponse(WebRequest request) throws IOException {
        // Log the URL of the request
        System.out.println(request.getUrl().toString());
        return super.getResponse(request);
    }

    public static void main(String[] args) throws IOException {
        try (WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
            // 配置WebClient选项
            webClient.getOptions().setJavaScriptEnabled(true);
            webClient.getOptions().setCssEnabled(true);
            webClient.getOptions().setRedirectEnabled(true);
            webClient.getOptions().setUseInsecureSSL(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.getCookieManager().setCookiesEnabled(true);
            webClient.setAjaxController(new AjaxController());

            // 设置请求拦截器
            webClient.setWebConnection(new RenderHTML(webClient));

            // 加载本地HTML文件
            HtmlPage page = webClient.getPage("file:///Users/derrickguo/work/project/project_java/analyze_demand_tool_maven/src/main/lib/algorix_us_adm.html");

            // 等待后台JavaScript执行完成，确保所有动态请求触发
            webClient.waitForBackgroundJavaScriptStartingBefore(100_000);
            webClient.waitForBackgroundJavaScript(100_000);
            // 额外等待所有JavaScript任务完成
            webClient.waitForJavaScriptJobs();
        }
    }
}

额外说明 #

• 确保使用的HTMLUnit版本是最新稳定版，旧版本可能存在JavaScript解析或请求拦截的bug。
• document.write会动态插入HTML内容，这些内容中的资源（图片、脚本）需要等待JavaScript执行后才会触发请求，必须在页面加载后等待足够时间。
• 如果仍有请求未捕获，可以适当延长等待时间（比如调整为200_000毫秒），或检查是否有跨域/SSL配置问题导致请求被拦截。

内容的提问来源于stack exchange，提问作者defalt1996