使用Pandas爬取Ballotpedia网页表格时出现ValueError报错求助

问题背景 #

使用Pandas read_html 爬取https://ballotpedia.org/Roger_Wicker页面时，触发ValueError，报错提示表头推断与实际表格行数不匹配。

报错代码 #

import pandas as pd
website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker")
print(website_df)

中文报错信息 #

Traceback (most recent call last):
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 700, in _next_line
    line = self._check_comments([self.data[self.pos]])[0]
IndexError: 列表索引越界

处理上述异常时引发新异常：

Traceback (most recent call last):
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 385, in _infer_columns
    line = self._next_line()
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 713, in _next_line
    raise StopIteration
StopIteration

该异常直接导致以下错误：

Traceback (most recent call last):
  File "c:\Users\legislators-current.py", line 15, in <module>
    website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker")
  File "C:\Users\miniconda3\lib\site-packages\pandas\util\_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\html.py", line 1205, in read_html
    return _parse(
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\html.py", line 1011, in _parse
    df = _data_to_frame(data=table, **kwargs)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\html.py", line 890, in _data_to_frame
    with TextParser(body, header=header, **kwargs) as tp:
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1876, in TextParser
    return TextFileReader(*args, **kwds)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1753, in _make_engine
    return mapping[engine](f, **self.options)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 122, in __init__
    ) = self._infer_columns()
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 395, in _infer_columns
    raise ValueError(
ValueError: 传入的header=[1,2]长度为2，但文件中仅包含2行内容

修复方案 #

核心原因 #

页面中存在结构异常的表格（如仅含表头无数据行、表头跨多行但内容不足），read_html默认的自动表头推断逻辑误判了表头行数，导致索引越界。

具体解决方法 #

1. 手动指定表头行
   强制设置表头为第一行，关闭自动推断：

   import pandas as pd
   website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker", header=0)
   print(website_df)
   

2. 过滤空表格
   爬取后剔除空的DataFrame，只保留有效表格：

   import pandas as pd
   website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker")
   valid_tables = [df for df in website_df if not df.empty]
   print(valid_tables)
   

3. 精准定位目标表格
   通过关键词匹配或索引指定，只爬取需要的表格（比如包含"Elections"的选举表格）：

   import pandas as pd
   # 匹配含指定关键词的表格
   website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker", match="Elections")
   print(website_df)
   

内容的提问来源于stack exchange，提问作者Jimmy C