Spring Boot Jsoup获取AFAD动态地震数据偶现500错误排查

我开发了一个Spring Boot Web API用于从AFAD网站获取地震数据。调用接口http://localhost:8080/api/v1/earthquakes/afad时，偶发两种结果：有时返回500内部错误（报错为java.lang.IndexOutOfBoundsException，触发于trs.remove(0)代码行），有时返回200正常结果。尝试使用HtmlUnit处理网页动态加载，但问题仍存在，寻求解决方案。

500错误信息 #

响应内容 #

{
    "timestamp": "2023-06-13T21:35:37.408+00:00",
    "status": 500,
    "error": "Internal Server Error",
    "path": "/api/v1/earthquakes/afad"
}

堆栈跟踪 #

java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100) ~[na:na]
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106) ~[na:na]
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302) ~[na:na]
    at java.base/java.util.Objects.checkIndex(Objects.java:385) ~[na:na]
    at java.base/java.util.ArrayList.remove(ArrayList.java:504) ~[na:na]
    at com.deprem.depremapi.utils.EarthQuakeDataParser.getEarthQuakeDataFromAfad(EarthQuakeDataParser.java:93) ~[classes/:na]
    at com.deprem.depremapi.service.EarthQuakeService.getEarthQuakeDataFromAfad(EarthQuakeService.java:31) ~[classes/:na]
    at com.deprem.depremapi.api.EarthquakeController.getAllEarthquakesForAfad(EarthquakeController.java:39)

200正常返回示例 #

[
    {
        "id": 1,
        "date": "2023-06-14 00:10:03",
        "timestamp": -1231544328,
        "latitude": 38.421,
        "longitude": 36.639,
        "depth": 7.0,
        "size": {
            "md": 0.0,
            "ml": 2.8,
            "mw": 0.0
        },
        "location": "Localtion Value",
        "afadId": "590282",
        "attribute": "ML"
    },
    ...
]

核心方法代码 #

public List<EarthquakeAfad> getEarthQuakeDataFromAfad() throws IOException, ParseException {

        log.info("EarthQuakeDataParser | getEarthQuakeDataFromAfad is working");

        List<EarthquakeAfad> earthquakeList = new ArrayList<>();

        Document doc = Jsoup.connect(Constants.AFAD_LINK).get();

        Elements trs = doc.select("tr");

        trs.remove(0);

        log.info("EarthQuakeDataParser | getEarthQuakeDataFromAfad | trs size : " + trs.size());

        for (int i = 0; i < trs.size(); i++) {

            Element tr = trs.get(i);
            Elements tds = tr.select("td");

            String earthquakeType = tds.get(4).text();

            EarthquakeAfad earthquake = EarthquakeAfad.builder()
                    .id((i+1))
                    .date(tds.get(0).text())
                    .timestamp((int) LocalDateTime.parse(tds.get(0).text(), DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"))
                            .atZone(ZoneId.systemDefault()).toInstant().toEpochMilli())
                    .latitude(Double.parseDouble(tds.get(1).text()))
                    .longitude(Double.parseDouble(tds.get(2).text()))
                    .depth(Double.parseDouble(tds.get(3).text()))
                    .size(Size.builder()
                            .md(earthquakeType.equals("MD") ? Double.parseDouble(tds.get(5).text()) : 0)
                            .ml(earthquakeType.equals("ML") ? Double.parseDouble(tds.get(5).text()) : 0)
                            .mw(earthquakeType.equals("MW") ? Double.parseDouble(tds.get(5).text()) : 0)
                            .build())
                    .location(tds.get(6).text())
                    .afadId(tds.get(7).text())
                    .attribute(earthquakeType)
                    .build();

            earthquakeList.add(earthquake);
        }

        return earthquakeList;
    }

HtmlUnit相关代码及依赖 #

Maven依赖 #

<dependency>
    <groupId>net.sourceforge.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>2.57.0</version>
</dependency>

加载网页代码 #

// 创建无头浏览器实例
WebClient webClient = new WebClient(BrowserVersion.CHROME);

// 启用JavaScript执行
webClient.getOptions().setJavaScriptEnabled(true);

// 禁用CSS提升性能（可选）
webClient.getOptions().setCssEnabled(false);

// 加载AFAD页面
HtmlPage page = webClient.getPage(Constants.AFAD_LINK);

// 等待JavaScript加载渲染完成
webClient.waitForBackgroundJavaScript(5000);

// 获取更新后的HTML内容
String html = page.asXml();

// 关闭WebClient释放资源
webClient.close();

// 使用Jsoup解析HTML
Document doc = Jsoup.parse(html);

Elements trs = doc.select("tr");

解决方案 #

1. 空列表检查 #

报错直接原因是trs.remove(0)时trs为空，必须先判断列表是否有元素：

Elements trs = doc.select("tr");
if (!trs.isEmpty()) {
    trs.remove(0);
} else {
    log.warn("未获取到AFAD网站的地震数据表格行");
    return Collections.emptyList();
}

2. 确保页面完全加载 #

HtmlUnit的waitForBackgroundJavaScript无法保证所有动态内容加载完成，可尝试：

• 延长等待时间至10秒：webClient.waitForBackgroundJavaScript(10000);
• 循环等待目标元素出现：

int attempts = 0;
while (page.getByXPath("//tr").isEmpty() && attempts < 5) {
    webClient.waitForBackgroundJavaScript(2000);
    attempts++;
}

3. 增强解析时的异常防护 #

解析TD元素时也可能出现索引越界，需检查元素数量：

Elements tds = tr.select("td");
if (tds.size() < 8) {
    log.warn("表格行数据不完整，跳过该行");
    continue;
}

4. 排查异常页面结构 #

AFAD网站可能偶尔返回维护页、加载失败页，导致无tr元素。可在获取doc后记录网页内容，分析异常场景下的页面结构，针对性处理。

内容的提问来源于stack exchange，提问作者Sercan Noyan Germiyanoğlu