如何正确定位XPath？网页爬取遇NullReferenceException问题求助

问题场景 #

尝试爬取指定篮球页面的四因素数据，运行C#代码时抛出空引用异常。

错误代码 #

string testurl = "https://www.basketball-reference.com/boxscores/202110190MIL.html";

var htmlDoc = new HtmlAgilityPack.HtmlDocument();
htmlDoc.LoadHtml(testurl);

var htmlNodes = htmlDoc.DocumentNode.SelectNodes("//*[@id=\"div_four_factors\"]");

int nodeCt = htmlNodes.Count;

异常信息 #

System.NullReferenceException: 'Object reference not set to an instance of an object.'

目标页面源码片段 #

<div class="table_container" id="div_four_factors">

<table class="suppress_all stats_table" id="four_factors" data-cols-to-freeze=",1">
<caption>Four Factors Table</caption>

<tr class="over_header">
     <th aria-label="" data-stat="" colspan="2" class=" over_header" ></th>
     <th aria-label="" data-stat="header_tmp" colspan="4" class=" over_header center" >Four Factors</th><th></th>
  </tr>
        
  <tr>
     <th aria-label="&amp;nbsp;" data-stat="team_id" scope="col" class=" poptip sort_default_asc left" data-tip="Team" >&amp;nbsp;</th>
     <th aria-label="Pace Factor" data-stat="pace" scope="col" class=" poptip right" data-tip="<b>Pace Factor</b>: An estimate of possessions per 48 minutes" >Pace</th>
     <th aria-label="Effective Field Goal Percentage" data-stat="efg_pct" scope="col" class=" poptip right" data-tip="<strong>Effective Field Goal Percentage</strong><br>This statistic adjusts for the fact that a 3-point field goal is worth one more point than a 2-point field goal." data-over-header="Four Factors" >eFG%</th>
     <th aria-label="Turnover Percentage" data-stat="tov_pct" scope="col" class=" poptip sort_default_asc right" data-tip="<b>Turnover Percentage</b><br>An estimate of turnovers committed per 100 plays." data-over-header="Four Factors" >TOV%</th>
     <th aria-label="Offensive Rebound Percentage" data-stat="orb_pct" scope="col" class=" poptip right" data-tip="<b>Offensive Rebound Percentage</b><br>An estimate of the percentage of available offensive rebounds a player grabbed while they were on the floor." data-over-header="Four Factors" >ORB%</th>
     <th aria-label="FT/FGA" data-stat="ft_rate" scope="col" class=" poptip right" data-tip="Free Throws Per Field Goal Attempt" data-over-header="Four Factors" >FT/FGA</th>
     <th aria-label="Offensive Rating" data-stat="off_rtg" scope="col" class=" poptip right" data-tip="<b>Offensive Rating</b><br>An estimate of points produced (players) or scored (teams) per 100 possessions" >ORtg</th>
  </tr>
  </thead>

解决步骤及修正代码 #

核心错误分析 #

1. LoadHtml方法误用：HtmlDocument.LoadHtml()接收的是HTML字符串，而非远程URL。直接传入URL会导致文档加载纯文本URL，而非页面HTML内容，自然找不到目标节点。
2. 未做空值判断：SelectNodes()在找不到匹配节点时会返回null，直接调用Count会触发空引用异常。

修正后的代码 #

using System.Net.Http;
using HtmlAgilityPack;

// 初始化HttpClient（建议全局单例）
using var httpClient = new HttpClient();
string testurl = "https://www.basketball-reference.com/boxscores/202110190MIL.html";

// 先获取页面HTML内容
string htmlContent = await httpClient.GetStringAsync(testurl);

var htmlDoc = new HtmlAgilityPack.HtmlDocument();
htmlDoc.LoadHtml(htmlContent);

// 查找目标节点，先判断是否为null
var htmlNodes = htmlDoc.DocumentNode.SelectNodes("//*[@id='div_four_factors']");
int nodeCt = htmlNodes?.Count ?? 0; // 空值合并运算符避免异常

// 后续处理四因素表格数据示例
if (htmlNodes != null)
{
    var tableNode = htmlNodes.FirstOrDefault()?.SelectSingleNode(".//table[@id='four_factors']");
    if (tableNode != null)
    {
        var rows = tableNode.SelectNodes(".//tbody/tr");
        foreach (var row in rows)
        {
            var teamLink = row.SelectSingleNode(".//td[@data-stat='team_id']/a");
            string teamName = teamLink?.InnerText;
            string pace = row.SelectSingleNode(".//td[@data-stat='pace']")?.InnerText;
            string efgPct = row.SelectSingleNode(".//td[@data-stat='efg_pct']")?.InnerText;
            // 按需提取其他字段...
        }
    }
}

关键说明 #

• XPath正确性：//*[@id='div_four_factors']是正确的，页面中确实存在该ID的容器节点。
• HttpClient优化：建议将HttpClient声明为全局单例，避免频繁创建销毁连接。
• 空值安全：所有节点查询后都要做null检查，从根源避免空引用异常。

内容的提问来源于stack exchange，提问作者Trey Balut