Html Agility Pack访问特定站点报无法创建SSL/TLS安全通道错误如何解决

根因分析 #

目标站点仅支持TLS 1.3协议，你当前仅配置了TLS 1.2，且默认请求缺少浏览器特征被站点WAF拦截，最终导致TLS握手失败，对应日志中的IllegalMessage错误码。

解决方案 #

1. 前置依赖检查 #

• .NET版本要求：.NET Framework 4.8+、.NET Core 3.0+ 原生支持TLS 1.3，低于该版本请先升级框架
• 操作系统要求：Windows Server 2022及以上版本默认支持TLS 1.3，低版本Windows Server需升级系统，或手动修改注册表开启TLS 1.3客户端支持：
  • 打开注册表路径：HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client
  • 新建DWORD值Enabled，赋值为1
  • 新建DWORD值DisabledByDefault，赋值为0
  • 重启系统生效

2. 代码适配修改 #

方案一：直接调整HtmlAgilityPack配置

将以下代码放在程序入口最开始执行，同时给HtmlWeb配置浏览器特征请求头：

// 全局配置TLS协议
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;
ServicePointManager.ServerCertificateValidationCallback = (sender, cert, chain, sslPolicyErrors) => true;

// 业务逻辑代码
var url = @"https://support.nedap-healthcare.com/topic/datamodellen-onsdb/";
var element = @"/html/body/div[2]/div/section[3]/div/div/div/div[2]/div[1]/div[2]/div[3]/div/ul";

HtmlWeb web = new HtmlWeb();
web.PreRequest = (HttpWebRequest request) =>
{
    // 添加浏览器特征头，绕过WAF拦截
    request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";
    request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8";
    request.Headers.Add("Accept-Language", "nl,en-US;q=0.7,en;q=0.3");
    request.Headers.Add("Accept-Encoding", "gzip, deflate, br");
    // 自动解压缩响应内容
    request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli;
    return true;
};

var doc = web.Load(url);
var HeaderNames = doc.DocumentNode.SelectNodes(element);

方案二：改用HttpClient拉取内容后解析

如果HtmlWeb内置的请求逻辑仍有问题，可改用更灵活的HttpClient拉取页面后再交给HtmlAgilityPack解析：

// 全局TLS配置和方案一一致
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;

var url = @"https://support.nedap-healthcare.com/topic/datamodellen-onsdb/";
var element = @"/html/body/div[2]/div/section[3]/div/div/div/div[2]/div[1]/div[2]/div[3]/div/ul";

// 初始化HttpClient
var handler = new HttpClientHandler
{
    ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => true,
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli
};
using var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36");
client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8");

// 拉取页面内容
var htmlContent = client.GetStringAsync(url).Result;
// 解析内容
var doc = new HtmlDocument();
doc.LoadHtml(htmlContent);
var HeaderNames = doc.DocumentNode.SelectNodes(element);

内容的提问来源于stack exchange，提问作者Marco Noordam