使用Pandas read_html读取SEC表单XML时遭遇ValueError：invalid literal for int() with base 10: '40%'的解决咨询

嘿，我完全懂你碰到的这个坑——pd.read_html底层依赖的HTML解析器（lxml/BeautifulSoup）默认会把rowspan和colspan属性当成整数来处理，但这份SEC表单里偏用了rowspan="40%"这种百分比值，直接触发了整数转换失败的错误。浏览器能正常渲染是因为它对这种非标准的rowspan值有容错处理，但pandas的解析器可没这么灵活。

你贴的源码片段里那个<td valign="top" rowspan="40%" colspan="50%">就是问题根源，pandas尝试把"40%"转成整数时直接炸了。下面给你几个实用的解决思路：

方案1：正则预处理内容，替换百分比属性 #

最简单的办法是先把响应内容里的百分比格式的rowspan/colspan替换成合理的整数值（比如1，因为这个单元格里是嵌套表格，实际不需要跨多行）：

import re
import requests
import pandas as pd

headers = {
    "User-Agent": "Alias (alias118@gmail.com)",
    "Accept-Encoding": "gzip, deflate",
    "Host": "www.sec.gov"
}

filing_url = 'https://data.sec.gov/Archives/edgar/data/320193/000032019323000048/xslF345X04/wf-form4_168064750462974.xml'
x = requests.get(filing_url, headers=headers)

if x.status_code != 200:
    print(f'Error loading xml for file:\n{filing_url}\nReason: {x.reason}')
else:
    # 预处理内容，替换百分比的rowspan/colspan为1
    cleaned_content = re.sub(r'rowspan="(\d+)%"', r'rowspan="1"', x.content.decode('utf-8'))
    cleaned_content = re.sub(r'colspan="(\d+)%"', r'colspan="1"', cleaned_content)
    
    try:
        tbls = pd.read_html(cleaned_content)
        # 后续处理表格数据
        print(f"成功解析到{len(tbls)}个表格")
    except Exception as e:
        print(f"解析出错: {e}")

方案2：用BeautifulSoup精细处理表格元素 #

如果正则替换不够灵活，你可以用BeautifulSoup遍历所有表格标签，针对性修改属性：

from bs4 import BeautifulSoup
import re
import requests
import pandas as pd

headers = {
    "User-Agent": "Alias (alias118@gmail.com)",
    "Accept-Encoding": "gzip, deflate",
    "Host": "www.sec.gov"
}

filing_url = 'https://data.sec.gov/Archives/edgar/data/320193/000032019323000048/xslF345X04/wf-form4_168064750462974.xml'
x = requests.get(filing_url, headers=headers)

if x.status_code != 200:
    print(f'Error loading xml for file:\n{filing_url}\nReason: {x.reason}')
else:
    soup = BeautifulSoup(x.content, 'html.parser')
    # 修正所有带百分比的rowspan属性
    for tag in soup.find_all(['td', 'th'], attrs={'rowspan': re.compile(r'\d+%')}):
        tag['rowspan'] = '1'
    # 修正所有带百分比的colspan属性
    for tag in soup.find_all(['td', 'th'], attrs={'colspan': re.compile(r'\d+%')}):
        tag['colspan'] = '1'
    
    try:
        tbls = pd.read_html(str(soup))
        print(f"成功解析到{len(tbls)}个表格")
    except Exception as e:
        print(f"解析出错: {e}")

方案3：直接解析XML（更推荐） #

其实你处理的是标准化的SEC Form4 XML文件，完全没必要绕路用HTML解析器。直接用XML解析工具提取数据会更稳定，还能避免HTML解析的各种兼容性问题：

import xml.etree.ElementTree as ET
import requests
import pandas as pd

headers = {
    "User-Agent": "Alias (alias118@gmail.com)",
    "Accept-Encoding": "gzip, deflate",
    "Host": "www.sec.gov"
}

filing_url = 'https://data.sec.gov/Archives/edgar/data/320193/000032019323000048/xslF345X04/wf-form4_168064750462974.xml'
x = requests.get(filing_url, headers=headers)

if x.status_code != 200:
    print(f'Error loading xml for file:\n{filing_url}\nReason: {x.reason}')
else:
    root = ET.fromstring(x.content)
    # 注意：SEC XML有命名空间，需要先处理
    ns = {'sec': 'http://www.sec.gov/edgar/document/form4'}
    transactions = root.findall('.//sec:transaction', namespaces=ns)
    
    columns = [
        'title', 'trade_date', 'execution_date', 'trade_code',
        'trade_code_v', 'shares_traded', 'acq_code', 'price',
        'shares_remaining', 'own_type', 'relationship'
    ]
    data = []
    
    for trans in transactions:
        row = {}
        # 提取交易日期
        trade_date = trans.find('sec:transactionDate/sec:value', namespaces=ns)
        row['trade_date'] = trade_date.text if trade_date else None
        # 提取交易股数
        shares_traded = trans.find('sec:transactionAmounts/sec:transactionShares/sec:value', namespaces=ns)
        row['shares_traded'] = shares_traded.text if shares_traded else None
        # 其他字段可以根据XML结构依次提取，这里给你做个示例
        row['title'] = root.find('sec:issuer/sec:issuerName', namespaces=ns).text
        data.append(row)
    
    df = pd.DataFrame(data, columns=columns)
    print(df.head())

至于你提到另一个文件能正常读取，只是因为那个文件里的rowspan/colspan都是整数值，刚好符合pandas解析器的预期而已。

备注：内容来源于stack exchange，提问作者Chris