使用C#爬虫部署到云服务时遭遇403 Forbidden错误求助

问题描述 #

使用C#和HTMLAgilityPack编写的网页爬虫，本地运行正常，但发布到Azure API服务或HostGator虚拟主机时，始终返回403 Forbidden错误，尝试多种方法仍无法解决。

相关代码 #

主爬虫代码 #

HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("https://antenati.cultura.gov.it/ark:/12657/an_ud18290200");
//string returnedResult = doc.DocumentNode.OuterHtml; //this shows a 403 forbidden error response when not running from localhost.
string ress = doc.DocumentNode.SelectSingleNode("//*[text()[contains(., 'manifestId:')]]").InnerText;

if (!string.IsNullOrEmpty(ress))
{
    string[] strPieces = ress.Split(new string[] { "manifestId:" }, StringSplitOptions.None);
    if (strPieces.Length >= 2)
    {
        WebClient wb = new WebClient();
        string manifestUrl = strPieces[1].Split(',')[0].Replace("'", "").Trim();
        wb.Headers.Add("origin", "https://antenati.cultura.gov.it");
        wb.Headers.Add("referer", "https://antenati.cultura.gov.it/");
        wb.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36");

        string result = wb.DownloadString(manifestUrl);
    }
}

测试代码（dotnetfiddle上运行返回403） #

using System;
using System.IO;
using System.Net;
                    
public class Program
{
    public static void Main()
    {
        string text = "";
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://antenati.cultura.gov.it/ark:/12657/an_ud18290200");
        
        //request.Proxy = new WebProxy("173.192.21.89", 80);

        request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0";
        request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8";
        //request.Connection = "keep-alive";
        request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
        //request.Headers.Add("Accept-Encoding", "gzip, deflate");
        request.Headers.Add("Upgrade-Insecure-Requests", "1");
        request.Headers.Add("Sec-Fetch-Dest", "document");
        request.Headers.Add("Sec-Fetch-Mode", "navigate");
        request.Headers.Add("Sec-Fetch-Site", "none");
        request.Headers.Add("Sec-Fetch-User", "?1");
        request.Headers.Add("Cache-Control", "max-age=0");      

        // Get the response.  
        WebResponse response = request.GetResponse();

        using (var sr = new StreamReader(response.GetResponseStream()))
        {
            text = sr.ReadToEnd();
        }

        Console.WriteLine(text);
    }
}

解决思路与方案 #

1. 完善请求头，模拟真实浏览器请求 #

目标网站可能通过检查请求头完整性识别爬虫，需补充以下关键设置：

• 启用压缩编码处理，添加Accept-Encoding并配置自动解压
• 补充DNT、Connection等常见浏览器头
• 确保Sec-*系列头与真实浏览器行为一致

修改后的HttpWebRequest请求头示例：

request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7";
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
request.Headers.Add(HttpRequestHeader.AcceptLanguage, "zh-CN,zh;q=0.9,en;q=0.8");
request.Headers.Add("DNT", "1");
request.Headers.Add("Connection", "keep-alive");
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli;

2. 维护会话Cookie #

目标网站可能通过Cookie验证请求合法性，需使用CookieContainer保留会话：

// 针对HttpWebRequest
CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;

// 针对HtmlWeb
HtmlWeb web = new HtmlWeb();
web.UseCookies = true;
web.CookieContainer = new CookieContainer();
HtmlDocument doc = web.Load("https://antenati.cultura.gov.it/ark:/12657/an_ud18290200");

3. 替换为HttpClient（推荐） #

WebClient是老旧API，HttpClient支持更多现代HTTP特性，替换示例：

var handler = new HttpClientHandler
{
    CookieContainer = new CookieContainer(),
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli
};

using (var httpClient = new HttpClient(handler))
{
    httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36");
    httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7");
    httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
    httpClient.DefaultRequestHeaders.Add("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8");
    httpClient.DefaultRequestHeaders.Add("DNT", "1");
    
    string html = await httpClient.GetStringAsync("https://antenati.cultura.gov.it/ark:/12657/an_ud18290200");
    // 后续HTML处理逻辑
}

4. 自定义HtmlWeb请求头 #

通过PreRequest事件为HtmlWeb补充完整请求头：

HtmlWeb web = new HtmlWeb();
web.PreRequest += (request) =>
{
    request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36";
    request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7";
    request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
    request.Headers.Add(HttpRequestHeader.AcceptLanguage, "zh-CN,zh;q=0.9,en;q=0.8");
    request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli;
    return true;
};
HtmlDocument doc = web.Load("https://antenati.cultura.gov.it/ark:/12657/an_ud18290200");

5. 检查IP是否被封禁 #

云服务或虚拟主机的IP可能被目标网站标记为爬虫IP段，可尝试：

• 使用代理IP发送请求
• 降低请求频率，添加随机访问延迟
• 联系目标网站申请合法访问权限（若提供官方API）

内容的提问来源于stack exchange，提问作者Jakal