使用HtmlUnit爬取移动端网站时动态HTML解析失败的问题排查

Let's work through your problem step by step—first we'll verify if the site truly uses dynamic content, then tweak your HtmlUnit configuration to ensure you're capturing all loaded content.

Step 1: Confirm if the Site Uses Dynamic JavaScript Rendering #

Before diving into code changes, let's confirm if the content you're missing is actually generated dynamically:

• Open Chrome DevTools (F12) and switch to the Device Toolbar (toggle mobile view)
• Set the device to an iPhone (match your User Agent's target, or a modern equivalent)
• Go to the Network tab, check "Disable cache", then refresh the page
• Right-click the page and select View Page Source—compare this to the content in the Elements tab. If the Elements tab has significantly more content than the raw page source, the site definitely relies on JS to render content.

Step 2: Fix Your HtmlUnit Configuration #

Your current setup has a few areas that might be preventing dynamic content from loading:

Update the User Agent #

The UA you're using is extremely outdated (iPhone OS 3.0—released in 2009!). Modern sites often block or limit functionality for legacy browsers. Swap it for a recent mobile UA, e.g.:

myMobileExplorerBuilder.setUserAgent("Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1");

Add JavaScript Wait Time #

HtmlUnit might be grabbing the page before JS finishes executing. Add a wait after loading the page to let background scripts complete:

HtmlPage page = client.getPage(url);
// Wait 5 seconds for JS to render content
client.waitForBackgroundJavaScript(5000);
// Optional: Wait for any pending JS that started before the 5-second window
client.waitForBackgroundJavaScriptStartingBefore(5000);

Clean Up Unnecessary Settings #

Mobile browsers don't use ActiveX, so disable that:

client.getOptions().setActiveXNative(false);

You might also want to re-enable CSS—some JS frameworks rely on CSS selectors to render content:

client.getOptions().setCssEnabled(true);

Enable Redirects and Cookies #

Many mobile sites use redirects and require cookies to load content properly:

client.getOptions().setRedirectEnabled(true);
client.getCookieManager().setCookiesEnabled(true);

Step 3: Debugging Tips #

If you're still missing content, use these tricks to diagnose the issue:

• Capture JavaScript Errors: Even if you disabled exception throwing, log JS errors to see if scripts are failing:

  client.setJavaScriptErrorListener(new JavaScriptErrorListener() {
      @Override
      public void scriptException(HtmlPage page, ScriptException scriptException) {
          System.err.println("JS Exception: " + scriptException.getMessage());
      }
      @Override
      public void timeoutError(HtmlPage page, long allowedTime, long executionTime) {
          System.err.println("JS Timeout: Took " + executionTime + "ms, allowed " + allowedTime + "ms");
      }
      // Implement other empty methods for the interface
      @Override public void malformedScriptURL(HtmlPage page, String url, MalformedURLException malformedURLException) {}
      @Override public void loadScriptError(HtmlPage page, URL scriptUrl, Exception exception) {}
  });
  

• Check Text Content: Sometimes asXml() doesn't show dynamic content correctly, but asText() might. Try:

  System.out.println(page.getBody().asText());
  

• Simulate Scrolling: Some content loads only when the user scrolls. Trigger a scroll and wait again:

  page.executeJavaScript("window.scrollTo(0, document.body.scrollHeight);");
  client.waitForBackgroundJavaScript(3000);
  

Modified Full Code Example #

Here's your code with all the recommended tweaks:

// Turn off annoying htmlunit warnings
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);

final BrowserVersion.BrowserVersionBuilder myMobileExplorerBuilder = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.FIREFOX_52);
// Use a modern iPhone User Agent
myMobileExplorerBuilder.setUserAgent("Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1");

WebClient client = new WebClient(myMobileExplorerBuilder.build());
client.getOptions().setCssEnabled(true);
client.getOptions().setJavaScriptEnabled(true);
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setActiveXNative(false);
client.getOptions().setRedirectEnabled(true);
client.getCookieManager().setCookiesEnabled(true);

// Optional: Add JS error logging for debugging
client.setJavaScriptErrorListener(new JavaScriptErrorListener() {
    @Override
    public void scriptException(HtmlPage page, ScriptException scriptException) {
        System.err.println("JS Exception: " + scriptException.getMessage());
    }
    @Override
    public void timeoutError(HtmlPage page, long allowedTime, long executionTime) {
        System.err.println("JS Timeout: Took " + executionTime + "ms, allowed " + allowedTime + "ms");
    }
    @Override public void malformedScriptURL(HtmlPage page, String url, MalformedURLException malformedURLException) {}
    @Override public void loadScriptError(HtmlPage page, URL scriptUrl, Exception exception) {}
});

String url = "https://mobile.bet365.es/";
HtmlPage page = client.getPage(url);

// Wait for JS to render content
client.waitForBackgroundJavaScript(5000);
client.waitForBackgroundJavaScriptStartingBefore(5000);

// Optional: Simulate scroll to load lazy content
page.executeJavaScript("window.scrollTo(0, document.body.scrollHeight);");
client.waitForBackgroundJavaScript(3000);

System.out.println(page.getBody().asXml());

Start with these changes—most likely the outdated UA or missing JS wait time is the root cause. If you still run into issues, check the JS error logs to see if specific scripts are failing to execute.

内容的提问来源于stack exchange，提问作者user1805792