如何用Java/HtmlUnit点击HtmlCell并获取动态加载的grdCPhist表格

问题描述 #

访问链接https://taxtest.navajocountyaz.gov/Pages/WebForm1.aspx?p=1&apn=205-27-014时，初始页面不存在grdCPhist表格；需先点击页面中的“Taxes”选项，再点击grdTaxHistory表格第5行第7列的CP单元格，页面才会动态加载出id为grdCPhist的表格。

使用Java/HtmlUnit编写的代码编译正常，但运行后始终无法找到该表格，输出grdCPHistory = null。

原代码 #

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
import com.gargoylesoftware.htmlunit.javascript.*;
import java.io.*;

public class ClickOnCell {

    public static void ClickOnCell () {
        try (final WebClient webClient = new WebClient()) {
            System.getProperties().put("org.apache.commons.logging.simplelog.defaultlog", "fatal");
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);

            webClient.getOptions().setCssEnabled(false);
            webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());
            webClient.setCssErrorHandler(new SilentCssErrorHandler());
            HtmlPage page = webClient.getPage("http://taxtest.navajocountyaz.gov/Pages/WebForm1.aspx?p=1&apn=205-27-014");
            webClient.waitForBackgroundJavaScriptStartingBefore(10000);
            page = (HtmlPage) page.getEnclosingWindow().getEnclosedPage();
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());
            HtmlTable grdTaxHistory = (HtmlTable) page.getElementById("grdTaxHistory");
            HtmlTableDataCell cpCell = (HtmlTableDataCell) grdTaxHistory.getCellAt(4,6);
            System.out.println("cpCell.getTextContent() = " + cpCell.getTextContent());
            cpCell.click();
            webClient.waitForBackgroundJavaScriptStartingBefore(1000000000);
            page = (HtmlPage) page.getEnclosingWindow().getEnclosedPage();
            HtmlTable grdCPHistory = (HtmlTable) page.getElementById("grdCPhist");
            System.out.println("grdCPHistory = " + grdCPHistory);
        }

        catch (Exception e) {
            System.out.println("Error: "+ e);
        }

    }

    public static void main(String[] args) {
        File file = new File("validParcelIDs.txt");
        ClickOnCell();
    }

}

原运行输出 #

WARNING: Obsolete content type encountered: 'text/javascript'.
Jan 13, 2023 5:51:29 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Jan 13, 2023 5:51:29 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Jan 13, 2023 5:51:30 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Jan 13, 2023 5:51:30 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Jan 13, 2023 5:51:30 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'text/javascript'.
cpCell.getTextContent() =                                                                      
                                         CP
                                            
grdCPHistory = null

修改方案 #

关键调整点 #

• 替换JS等待方法：使用webClient.waitForBackgroundJavaScript(10000)替代waitForBackgroundJavaScriptStartingBefore，该方法会等待指定时长内的后台JavaScript执行完毕，更适配动态内容加载场景。
• 正确处理点击后的页面：cpCell.click()会返回点击操作后的页面对象（或更新当前页面），直接使用该返回值，避免重复从窗口获取页面导致的内容不一致。
• 添加显式等待逻辑：循环等待直到grdCPhist表格出现，确保AJAX加载完成后再尝试获取元素。
• 启用CSS支持：部分动态渲染依赖CSS规则，启用CSS可提升页面渲染的准确性。

修改后的完整代码 #

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
import com.gargoylesoftware.htmlunit.javascript.*;
import java.io.*;

public class ClickOnCell {

    public static void ClickOnCell() {
        try (final WebClient webClient = new WebClient()) {
            // 配置日志级别，减少冗余输出
            System.getProperties().put("org.apache.commons.logging.simplelog.defaultlog", "fatal");
            
            // WebClient基础配置
            webClient.getOptions().setThrowExceptionOnScriptError(false);
            webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
            webClient.getOptions().setJavaScriptEnabled(true); // 明确启用JS
            webClient.getOptions().setCssEnabled(true); // 启用CSS，适配页面渲染
            webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());
            webClient.setCssErrorHandler(new SilentCssErrorHandler());

            // 访问目标页面
            HtmlPage page = webClient.getPage("https://taxtest.navajocountyaz.gov/Pages/WebForm1.aspx?p=1&apn=205-27-014");
            // 等待初始页面的JS执行完成
            webClient.waitForBackgroundJavaScript(10000);
            page = (HtmlPage) page.getEnclosingWindow().getEnclosedPage();

            // 获取TaxHistory表格并定位CP单元格
            HtmlTable grdTaxHistory = (HtmlTable) page.getElementById("grdTaxHistory");
            HtmlTableDataCell cpCell = (HtmlTableDataCell) grdTaxHistory.getCellAt(4, 6);
            System.out.println("cpCell.getTextContent() = " + cpCell.getTextContent().trim());

            // 点击CP单元格，获取更新后的页面
            page = cpCell.click();
            // 等待点击后的AJAX请求和JS渲染完成
            webClient.waitForBackgroundJavaScript(15000);

            // 显式等待grdCPhist表格加载完成，最多等待20秒
            long startTime = System.currentTimeMillis();
            HtmlTable grdCPHistory = null;
            while (grdCPHistory == null && (System.currentTimeMillis() - startTime) < 20000) {
                grdCPHistory = (HtmlTable) page.getElementById("grdCPhist");
                Thread.sleep(500); // 每500毫秒检查一次
            }

            // 输出结果
            if (grdCPHistory != null) {
                System.out.println("成功获取grdCPhist表格，行数：" + grdCPHistory.getRowCount());
                // 可添加表格内容遍历逻辑
            } else {
                System.out.println("grdCPHistory = null，表格未加载");
            }
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {
        ClickOnCell();
    }
}

编译运行命令 #

javac -classpath ".:/opt/htmlunit_2.69.0/*" ClickOnCell.java
java -classpath ".:/opt/htmlunit_2.69.0/*" ClickOnCell

内容的提问来源于stack exchange，提问作者God's Gift To Java