使用JSOUP爬取网页报SSL错误，仅单站点异常是否为站点专属问题？

我日常使用Jsoup执行网页爬取作业均正常，本次尝试爬取站点https://www.strack.de/de/shop/?idm=1162&d=1&idmp=94&spb=MTQ7NzQ7MTI0OzEyMzY7 下的多个页面时均出现SSL相关错误，其余站点爬取不受影响。
报错为SSL握手失败，完整异常栈如下：

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target at
sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at
sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at
sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) at
sun.security.ssl.Handshaker.process_record(Handshaker.java:914) at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at
sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at
org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:732)
at
org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:707)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:297)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:286) at
scrapetestforstack.de.ScrapeTestForStackDe.main(ScrapeTestForStackDe.java:81)
Caused by: sun.security.validator.ValidatorException: PKIX path
building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target at
sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
at
sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260) at
sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
... 15 more Caused by:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target at
sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at
sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at
sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
... 21 more
C:\Users\LeonardDME\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53:
Java returned: 1 BUILD FAILED (total time: 0 seconds)

我已尝试升级Java到最新版本、重启设备，还按照现有方案尝试通过Firefox导出站点证书，但当前浏览器版本未找到对应证书导出入口，上述操作均未解决问题。我在今年2-3月还可正常爬取该站点，怀疑是站点新增了反爬机制，核心爬取代码如下：

//Phase 3 Scrape The URL for Urls
Document doc = Jsoup.connect(URL).get();

title = doc.title();

TitleFixer = title.replaceAll(" ", "");
title = TitleFixer.replaceAll("|", "");
TitleFixer = title.replaceAll("|", "");
title = TitleFixer.replaceAll(";", "");

//Set file writing stuff 1
GimmeAName = ("C:\\Users\\LeonardDME\\Documents\\NetBeansProjects\\ScrapeTestForStackDe\\Urls\\" + title + ".csv");
File f = new File(GimmeAName);
FileWriter fw = new FileWriter(f);
PrintWriter out = new PrintWriter(fw);
StuffToWrite = URLArray[counter];
// fetch the document over HTTP
Elements spangrabbers = doc.getElementsByClass("art_orginal_preis142790");
for (Element spangrab : spangrabbers)
{
    holder2 = spangrab.text();
    SpanHolderArray[SpanHolderCounter] = holder2;
    SpanHolderCounter++;
}
// get all links in page
Elements links = doc.select("a[href]");
for (Element link : links) {
    // get the value from the href attribute
    checker = link.attr("href");
    if (checker.contains("http"))
    {
    }
    else if(checker.contains("javascript"))
    {
    }
    else if(checker.contains("style"))
    {
    }
    else
    {
        counter++;

        if(LinkContorter == null && LinkContorter.isEmpty())
        {
            //do nothing
        }
        else
        {
            System.out.println(LinkContorter);
            out.print(LinkContorter);
            out.print(",");
            out.print("\n");
            //Flush the output to the file
            out.flush();
        }
    }
}
System.out.println(counter);
//Close the Print Writer
out.close();
//Close the File Writer
fw.close();

原因判定 #

该问题并非站点反爬机制导致，核心原因是目标站点近期更新了SSL证书链，你使用的Java环境信任库中缺失对应根证书/中级证书，无法完成证书合法性校验，所以触发PKIX路径构建失败的报错。可正常访问其他HTTPS站点是因为那些站点的证书对应根证书已内置在Java默认信任库中。

解决方案 #

• 方案1：快速跳过SSL校验（仅推荐测试场景使用，生产环境存在安全风险）
  修改Jsoup连接配置，直接关闭证书校验即可：

Document doc = Jsoup.connect(URL)
    .sslSocketFactory(org.jsoup.helper.SSLSocketFactoryBuilder.create().trustAllCertificates().build())
    .validateTLSCertificates(false)
    .get();

• 方案2：导入证书到Java信任库（长期稳定方案，安全合规）

1. 使用Chrome浏览器访问目标站点，点击地址栏左侧锁图标->选择「证书」->切换到「详细信息」标签->点击「复制到文件」，按向导导出CER格式的根证书、中级证书
2. 找到你正在使用的JDK安装路径，进入jre/lib/security目录，执行导入命令：
   keytool -import -alias strack_de -keystore cacerts -file 你导出的证书本地路径.cer
   Java默认信任库cacerts的默认密码为changeit，导入完成后重启你的Java进程/IDE即可生效。

• 方案3：自定义信任管理器（无需修改JDK配置，适合权限受限场景）
  自己实现一个信任所有合法域名证书的X509TrustManager，绑定到Jsoup的SSL上下文即可，不需要修改JDK层面的信任库配置。

内容的提问来源于stack exchange，提问作者InternetMAn