C#网页爬取异常：Get请求无法获取网页表格数据

我正在尝试使用C#爬取某网站表格中的条目，但发送Get请求后，响应字符串中并未包含网页显示的那些表格条目。目标网站地址：https://www.services-rte.com/en/view-data-published-by-rte/downtime-of-generation-resources.html。我猜测需要发送Post请求来加载表格，但无法确定具体的请求参数，若我的判断有误请指正。

以下是我的代码：

static async void GetEntries()
{
    var services = new ServiceCollection();
    services.AddHttpClient();
    var serviceProvider = services.BuildServiceProvider();
    var httpClientFactory = serviceProvider.GetService<IHttpClientFactory>();
    var client = httpClientFactory.CreateClient();

    string response = string.Empty;
    try
    {
        response = await client.GetStringAsync("https://www.services-rte.com/en/view-data-published-by-rte/downtime-of-generation-resources.html");
    }
    catch
    {
        Console.WriteLine("Site not found.");
        return;
    }

    var parser = new HtmlParser();
    var document = parser.ParseDocument(response);

    string content = string.Empty;
    for (int i = 1; i <= 20; i++)
    {
        try
        {
            Console.WriteLine(i);
            content = document.QuerySelector($"#wrapper > div > div > div.c-editorial-page__container > div.c-editorial-page__content > ctx-remit-generation-unavailability > cortex-remit-generation-unavailability-table > cortex-table > div > div.ctx__table_content > cortex-table-row:nth-child({i})").TextContent;
        }
        catch
        {
            Console.WriteLine($"CSS selector not found for {i}.");
            continue;
        }

        Console.WriteLine(content);
        Console.WriteLine("NEW");
    }
}

报错行：

content = document.QuerySelector($"#wrapper > div > div > div.c-editorial-page__container > div.c-editorial-page__content > ctx-remit-generation-unavailability > cortex-remit-generation-unavailability-table > cortex-table > div > div.ctx__table_content > cortex-table-row:nth-child({i})").TextContent;

错误信息：对象引用未设置为对象的实例

原因分析 #

你遇到的问题是因为目标网站的表格内容是前端JavaScript动态加载的，不是页面初始HTML的一部分。直接发送Get请求获取的只是页面骨架，表格数据是后续通过AJAX请求加载的，所以你的HTML解析器找不到对应的元素。

具体解决步骤 #

1. 找到真实的数据接口
   打开浏览器开发者工具（F12），切换到「网络」标签页，刷新页面后筛选「XHR/Fetch」请求，能看到表格数据来自一个Post请求，接口地址是：https://www.services-rte.com/api/v1/remit/generation-unavailability/search

2. 构造Post请求参数
   该接口需要的请求体（JSON格式）大致如下，可根据需求调整分页、时间范围等参数：

{
  "size": 20,
  "page": 0,
  "sort": [
    {
      "field": "start_date",
      "direction": "desc"
    }
  ],
  "filters": {
    "start_date": {
      "from": "2024-01-01T00:00:00.000Z",
      "to": "2024-12-31T23:59:59.999Z"
    }
  }
}

3. 修改C#代码直接请求数据接口
   不需要解析HTML，直接请求API获取JSON数据，再反序列化为实体类处理即可：

using System;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;

// 定义对应数据结构的实体类
public class GenerationUnavailabilityResponse
{
    public int Total { get; set; }
    public GenerationUnavailabilityItem[] Items { get; set; }
}

public class GenerationUnavailabilityItem
{
    [JsonProperty("resource_name")]
    public string ResourceName { get; set; }
    [JsonProperty("start_date")]
    public string StartDate { get; set; }
    [JsonProperty("end_date")]
    public string EndDate { get; set; }
    // 其他字段根据接口返回的JSON结构补充
}

static async Task GetEntries()
{
    var services = new Microsoft.Extensions.DependencyInjection.ServiceCollection();
    services.AddHttpClient();
    var serviceProvider = services.BuildServiceProvider();
    var httpClientFactory = serviceProvider.GetService<IHttpClientFactory>();
    var client = httpClientFactory.CreateClient();

    // 设置请求头，模拟浏览器请求
    client.DefaultRequestHeaders.Add("Accept", "application/json");
    client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36");

    // 构造请求体
    var requestBody = new
    {
        size = 20,
        page = 0,
        sort = new[]
        {
            new { field = "start_date", direction = "desc" }
        },
        filters = new
        {
            start_date = new
            {
                from = "2024-01-01T00:00:00.000Z",
                to = DateTime.UtcNow.ToString("yyyy-MM-ddTHH:mm:ss.fffZ")
            }
        }
    };

    string jsonBody = JsonConvert.SerializeObject(requestBody);
    var content = new StringContent(jsonBody, Encoding.UTF8, "application/json");

    HttpResponseMessage response;
    try
    {
        response = await client.PostAsync("https://www.services-rte.com/api/v1/remit/generation-unavailability/search", content);
        response.EnsureSuccessStatusCode();
    }
    catch (HttpRequestException ex)
    {
        Console.WriteLine($"请求失败: {ex.Message}");
        return;
    }

    string responseJson = await response.Content.ReadAsStringAsync();
    var data = JsonConvert.DeserializeObject<GenerationUnavailabilityResponse>(responseJson);

    // 处理返回的数据
    foreach (var item in data.Items)
    {
        Console.WriteLine($"资源名称: {item.ResourceName}");
        Console.WriteLine($"开始时间: {item.StartDate}");
        Console.WriteLine($"结束时间: {item.EndDate}");
        Console.WriteLine("---");
    }
}

注意事项 #

• 确保项目中安装了Newtonsoft.Json和Microsoft.Extensions.Http NuGet包
• 接口参数（如时间范围、分页）可按需调整，避免请求过多数据触发反爬限制
• 若请求失败，可尝试添加Referer等更多请求头，进一步模拟真实浏览器行为

内容的提问来源于stack exchange，提问作者MegaPaul