使用HTTP代理通过HttpClient爬取HTTPS网站遇403/400错误求助

我尝试使用hidemy.name提供的HTTP代理，通过HttpClient爬取HTTPS网站（目标网站就是该代理列表页）。发现爬取HTTP网站正常，但所有HTTPS网站均爬取失败，返回400或403错误。曾尝试使用HTTPS代理，但当前.NET Core环境暂不支持HTTPS代理。以下是我的C#代码：

using System.Net;
using System.Web;
using HtmlAgilityPack;

internal sealed class TestProxy {
        private HttpClientHandler _htttpClientHandler;
        private HttpClient _client;

        internal Test() {
            System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

            var proxy = new WebProxy
            {
                Address = new Uri($"http://173.82.153.196:16781"), // This proxy is working as of 12/13/2022
                BypassProxyOnLocal = true,
                UseDefaultCredentials = false,
                Credentials = System.Net.CredentialCache.DefaultNetworkCredentials
            };

            this._htttpClientHandler = new HttpClientHandler
            {
                UseProxy = true,
                UseCookies = true,
                Proxy = proxy,
                DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials
            };

            _htttpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

            this._client = new HttpClient(this._htttpClientHandler, true);
       }

 private async Task<HtmlDocument> LoadPage(string url)
        {
            this._client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0");

            var stream = await this._client.GetStreamAsync(url);
            using (var reader = new StreamReader(stream))
            {
                string html = reader.ReadToEnd();
                var doc = new HtmlDocument();
                doc.LoadHtml(html);
                return doc;
            }
        }

}

报错信息：

System.Private.CoreLib.dll中发生了'System.Net.Http.HttpRequestException'类型的异常，但未在用户代码中处理: '响应状态码不表示成功: 403 (Forbidden)。'

1. 移除冗余的代理凭证配置 #

公开代理通常不需要Windows域环境的默认凭证，多余的凭证携带反而可能触发目标网站的拦截规则。修改代理和Handler的配置：

var proxy = new WebProxy
{
    Address = new Uri($"http://173.82.153.196:16781"),
    BypassProxyOnLocal = true,
    UseDefaultCredentials = false
    // 移除Credentials赋值
};

this._htttpClientHandler = new HttpClientHandler
{
    UseProxy = true,
    UseCookies = true,
    Proxy = proxy
    // 移除DefaultProxyCredentials赋值
};

2. 避免重复添加请求头 #

每次调用LoadPage都添加User-Agent会导致请求头中出现多个相同字段，部分网站会因此拒绝请求。将请求头初始化移到构造函数中：

internal Test() {
    // ... 其他初始化代码 ...
    this._client = new HttpClient(this._htttpClientHandler, true);
    // 仅添加一次User-Agent
    this._client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0");
}

private async Task<HtmlDocument> LoadPage(string url)
{
    // 移除此处的User-Agent添加代码
    var stream = await this._client.GetStreamAsync(url);
    // ... 后续代码不变 ...
}

3. 补充浏览器标准请求头 #

模拟真实浏览器的请求头组合，降低被识别为爬虫的概率：

internal Test() {
    // ... 其他初始化代码 ...
    this._client.DefaultRequestHeaders.Accept.ParseAdd("text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8");
    this._client.DefaultRequestHeaders.AcceptLanguage.ParseAdd("en-US,en;q=0.5");
    this._client.DefaultRequestHeaders.Referrer = new Uri("https://hidemy.name/en/proxy-list/");
}

4. 验证代理有效性 #

公开代理生命周期极短，你标注的代理可能已失效。使用前可先通过简单请求验证连通性，比如发送GET请求到http://httpbin.org/ip确认代理是否正常转发IP。

5. HTTPS代理替代方案 #

如果必须使用HTTPS代理，可尝试升级到.NET 6+版本（该版本对HTTPS代理支持有所优化），或使用第三方库如ProxyKit实现代理转发逻辑。

内容的提问来源于stack exchange，提问作者emhsmath