如何用Python库完整解析SEC的13-F表单为SGML格式？

问题背景 #

目标是解析SEC网站上的13-F表单，提取所有infoTable元素。首先通过以下代码获取目标文件数据：

from urllib.request import Request, urlopen
import gzip

url = "https://www.sec.gov/Archives/edgar/data/1067983/000095012324011775/0000950123-24-011775.txt"
req = Request(
    url=url,
    headers={'User-Agent': 'xxxx@gmail.com',
             "Accept-Encoding":"gzip, deflate",
             'Host': 'www.sec.gov'}
)
webpage = urlopen(req).read()
content = gzip.decompress(webpage)
data = content.decode('utf-8')

常规XML解析库的报错情况 #

尝试使用多个XML解析库解析时均失败，报错指向第17行第52列格式问题：

使用minidom报错 #

from xml.dom import minidom   
xmldoc = minidom.parseString(data)

报错信息：

Traceback (most recent call last):     
  File "<stdin>", line 1, in <module>    
  File "/usr/lib/python3.11/xml/dom/minidom.py", line 2000, in parseString
    return expatbuilder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/dom/expatbuilder.py", line 925, in parseString
    return builder.parseString(string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/dom/expatbuilder.py", line 223, in parseString
  parser.Parse(string, True)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 17, column 52

使用xml.etree报错 #

import xml.etree.ElementTree as ET
tree = ET.fromstring(data)

报错信息：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1338, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 17, column 52

使用lxml.etree报错 #

from lxml import etree
tree = etree.fromstring(data)

报错信息：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/lxml/etree.pyx", line 3306, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1995, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1875, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1105, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 743, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 672, in lxml.etree._raiseParseError
  File "<string>", line 17
lxml.etree.XMLSyntaxError: xmlParseEntityRef: no name, line 17, column 53

临时解决方案：正则提取+XML解析 #

通过正则表达式提取informationTable节点内容后，再用XML解析库处理：

正则提取核心内容 #

import re
pattern = re.compile(r"<informationTable.*?>.*?</informationTable>", flags=re.DOTALL)
data_str = re.findall(pattern, data)[0]

用minidom解析提取结果 #

from xml.dom import minidom 
dom = minidom.parseString(data_str)
dlist = dom.getElementsByTagName('infoTable')    
result = []
for item in dlist:
    emp_result = {}
    for child in item.childNodes:
        if child.nodeType == minidom.Node.ELEMENT_NODE:
            emp_result[child.tagName] = child.firstChild.data
    result.append(emp_result)

用pandas解析提取结果 #

import pandas as pd
from io import StringIO  
df = pd.read_xml(StringIO(data_str))

更高效的SGML直接解析方法 #

SEC的13-F表单本质是SGML格式，而非严格XML，因此可以使用专门的SGML解析库直接处理整个文件，推荐使用sgmllib3k（Python3兼容版的sgmllib）：

1. 先安装库：

pip install sgmllib3k

2. 自定义解析类提取infoTable元素：

from sgmllib3k import SGMLParser

class InfoTableParser(SGMLParser):
    def __init__(self):
        super().__init__()
        self.in_info_table = False
        self.current_info = {}
        self.current_tag = None
        self.info_tables = []

    def start_element(self, tag, attrs):
        if tag == 'infoTable':
            self.in_info_table = True
            self.current_info = {}
        elif self.in_info_table:
            self.current_tag = tag

    def end_element(self, tag):
        if tag == 'infoTable':
            self.in_info_table = False
            self.info_tables.append(self.current_info)
        elif self.in_info_table:
            self.current_tag = None

    def handle_data(self, data):
        if self.in_info_table and self.current_tag:
            # 清理空白字符
            cleaned_data = data.strip()
            if cleaned_data:
                self.current_info[self.current_tag] = cleaned_data

# 初始化解析器并解析数据
parser = InfoTableParser()
parser.feed(data)
parser.close()

# 获取所有infoTable数据
all_info_tables = parser.info_tables
# 可以转为DataFrame查看
import pandas as pd
df = pd.DataFrame(all_info_tables)

这种方法无需提前提取片段，直接处理整个SGML文件，更符合文件本身的格式规范，也避免了正则表达式可能出现的匹配问题。

内容的提问来源于stack exchange，提问作者showkey