Java 11使用Jsoup 1.14.2爬取HTTPS页面报SSLHandshakeException问题

这是我首次在此提问，如有操作不当的地方请及时告知。
我使用Java 11编写了少量代码，搭配Jsoup (1.14.2)从某网店爬取信息，由于该网店的数据有多页，我使用循环获取所有需要的URL。
以下是我的简化示例代码：

for (int i = 1; i < 36; i++) {
    String url = ("https://www.play-in.com/rachat/hotlist/magic?p=" + i);

    try {
        doc = Jsoup.connect(url).get();
    } catch (Exception e) {
        logger.info("Impossible de récuppérer les éléments de la page " + i + " : " + e);
    }

    // 此处编写HTML解析逻辑，返回对象数组
}

运行程序时得到如下输出：

[main] INFO  service.MagicBazarReader  -  Failed to get data from page 2 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[main] INFO  service.MagicBazarReader  -  Failed to get data from page 3 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[main] INFO  service.MagicBazarReader  -  Failed to get data from page 4 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[main] INFO  service.MagicBazarReader  -  Failed to get data from page 5 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[main] INFO  service.MagicBazarReader  -  Failed to get data from page 6 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[...]
[main] INFO  ISmellProfits  - Number of result after HTML parsing : 24

以此类推。首次get()请求总是成功，我可以正常处理返回结果，但之后多次调用Jsoup.connect()时就会出现问题。
由于我请求的是HTTPS链接，我首先想到是证书问题，尝试了相关解决方案，但没有效果。如果确实需要证书的话，我应该首次也无法访问该URL才对，不过我对这个领域了解不多，可能判断有误。
第二个思路是使用parallel stream，代码如下：

List <String> links = new ArrayList<>();
for (int i = 1; i < 36; i++) {
    String url = ("https://www.play-in.com/rachat/hotlist/magic?p=" + i);
    links.add(url);
}

links.parallelStream().forEach(link -> {
    Document doc = new Document("");
    try {
        doc = Jsoup.connect(link).get();

        // 此处编写HTML解析逻辑，返回对象数组
    } catch (Exception e) {
        logger.info("Impossible de récuppérer les éléments de la page " + link.substring(link.length() - 2) + " : " + e);
    }
});

运行效果更好，但仍不完善，报错如下：

[ForkJoinPool.commonPool-worker-17] INFO  service.MagicBazarReader  - Failed to get data from page 12 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-23] INFO  service.MagicBazarReader  -  Failed to get data from page 30 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-9] INFO  service.MagicBazarReader  -  Failed to get data from page 5 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-27] INFO  service.MagicBazarReader  -  Failed to get data from page 35 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[main] INFO  service.MagicBazarReader  -  Failed to get data from page 22 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-3] INFO  service.MagicBazarReader  -  Failed to get data from page 1 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-7] INFO  service.MagicBazarReader  -  Failed to get data from page 2 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-21] INFO  service.MagicBazarReader  -  Failed to get data from page 17 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[ForkJoinPool.commonPool-worker-5] INFO  service.MagicBazarReader  -  Failed to get data from page 31 : javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
[...]
[main] INFO  ISmellProfits  - Number of result after HTML parsing : 286

因此HTML解析后得到的结果多了很多，但结果并不稳定，每次运行得到的数量都不同，且仍会抛出SSLHandshakeException。
我已经没有其他思路了，想请教各位是什么原因导致该异常被抛出。我刚接触Jsoup，对它还不太熟悉，我猜测可能是Jsoup同一时间只能建立一个连接，循环在第一个连接关闭前就发起了新连接导致的问题。

核心原因 #

你遇到的不是Jsoup的连接限制问题，也不是证书有效性问题，核心原因是目标站点的反爬机制触发了TLS连接主动断开：

• 短时间内发起大量无伪装的请求，站点识别到爬虫特征后会直接中断TLS握手流程，返回handshake_failure错误
• 默认的parallelStream并发数过高，进一步提高了触发限流的概率，所以才会出现结果不稳定的情况

解决方案 #

1. 优先使用单线程+请求间隔的方案 #

单线程添加请求间隔是最稳妥的规避反爬的方式，同时添加请求头模拟真实浏览器行为，示例代码如下：

// 模拟Chrome浏览器请求头
String userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";
for (int i = 1; i < 36; i++) {
    String url = "https://www.play-in.com/rachat/hotlist/magic?p=" + i;
    Document doc = null;
    try {
        doc = Jsoup.connect(url)
                .userAgent(userAgent)
                .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8")
                .header("Accept-Language", "fr-FR,fr;q=0.8,en-US;q=0.5,en;q=0.3")
                .header("Connection", "keep-alive")
                .timeout(10000)
                .get();
        // 此处编写解析逻辑
    } catch (Exception e) {
        logger.info("无法获取第{}页数据：{}", i, e.getMessage());
        // 出错后等待3秒重试一次
        Thread.sleep(3000);
        // 重试逻辑可根据需求自行补充
    }
    // 每次请求后间隔1.5秒再发起下一次请求
    Thread.sleep(1500);
}

2. 并行请求控制并发数 #

如果需要提高爬取效率，不要使用默认的parallelStream，自行创建并发数为2-3的线程池即可，避免并发过高触发限流。

3. 额外优化项 #

• 升级Jsoup到最新稳定版，1.14.2版本存在部分HTTPS适配问题，新版本已修复
• 如果仍出现TLS握手错误，可手动指定TLS版本适配站点要求：

SSLContext sslContext = SSLContext.getInstance("TLSv1.3");
sslContext.init(null, null, new SecureRandom());
SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();

// 调用时传入自定义的SSLSocketFactory
doc = Jsoup.connect(url)
        .sslSocketFactory(sslSocketFactory)
        // 其余配置保持不变
        .get();

内容的提问来源于stack exchange，提问作者Pulco