Replit中Python转换Endnote XML大文件报错：无效令牌

错误信息 #

解析30-50条记录的大文件时出现如下XML解析错误：

File "main.py", line 12, in convert_endnote_to_word
    root = ET.fromstring(endnote_xml_str)
File "/nix/store/2vm88xw7513h9pyjyafw32cps51b0ia1-python3-3.8.12/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 2248

现有代码 #

import xml.etree.ElementTree as ET
def convert_endnote_to_word(endnote_xml_str):
    word_xml_str = '<?xml version="1.0" encoding="utf-8"?>\n'
    word_xml_str += '<b:Sources xmlns:b="http://schemas.openxmlformats.org/officeDocument/2006/bibliography" '
    word_xml_str += 'xmlns="http://schemas.openxmlformats.org/officeDocument/2006/bibliography" SelectedStyle="">\n' 
    root = ET.fromstring(endnote_xml_str)
    for record in root.findall('./records/record'):
        source = {}
        word_xml_str += '  <b:Source>\n'
        year = record.find('./pub-dates/date').text
        source['year'] = year
        word_xml_str += f'    <b:Year>{source["year"]}</b:Year>\n'  
        source['ref_type'] = record.find('./ref-type').get('name').replace(" ", "")
        word_xml_str += f'    <b:SourceType>{source["ref_type"]}</b:SourceType>\n'
        title = record.find('./titles/title').text
        source['title'] = title
        word_xml_str += f'    <b:Title>{source["title"]}</b:Title>\n'        
        authors = []
        for author in record.findall('./contributors/authors/author'):
            authors.append(author.text)
        source['authors'] = authors
        convert_authors_and_year(source)
        word_xml_str += f'    <b:Tag>{source["tag"]}</b:Tag>\n'
        word_xml_str += f'    <b:Author>\n{source["authors_xml"]}    </b:Author>\n'  
        source['url'] = record.find('./urls/related-urls/url').text
        word_xml_str += f'    <b:URL>{source["url"]}</b:URL>\n'
        word_xml_str += '  </b:Source>\n'
    word_xml_str += '</b:Sources>'
    return word_xml_str

def convert_authors_and_year(source): 
    authors = source['authors']
    year = source['year']
    last_names = [author.split()[-1] for author in authors]
    tag = f'{", ".join(last_names)} ({year})'
    source['tag'] = tag
    source['authors_xml'] = ''
    source['authors_xml'] += f'      <b:Author>\n        <b:NameList>\n'
    for author in authors:
        source['authors_xml'] += f'          <b:Person>\n'
        source['authors_xml'] += f'            <b:Last>{author.split(" ")[-1]}</b:Last>\n'
        source['authors_xml'] += f'            <b:First>{author.split(" ")[0]}</b:First>\n'
        source['authors_xml'] += '          </b:Person>\n'
    source['authors_xml'] += '        </b:NameList>\n      </b:Author>\n'

def save_word_file(endnote_xml_str,word_filepath: str='word_xml_file.xml'):
    word_xml_str = convert_endnote_to_word(endnote_xml_str)
    with open(word_filepath, 'w') as f:
      f.write(word_xml_str)

def convert_file(endnote_filepath: str='endnote_xml_file.xml', word_filepath: str='word_xml_file.xml'):
  with open(endnote_filepath, 'r') as f:
    endnote_xml_str = f.read()
  save_word_file(endnote_xml_str, word_filepath)

问题原因 #

1. XML非法字符：大文件中存在未转义的XML特殊字符（如&、<、>），小文件未包含这类字符所以正常解析，大文件触发解析错误。
2. 编码问题：读取文件时未指定编码，导致部分字符解析为乱码，被识别为无效token。
3. 内存加载限制：ET.fromstring()将整个XML字符串加载到内存，大文件可能出现隐性的字符串截断或编码异常。
4. 字段缺失未处理：部分记录可能缺失year、url等字段，直接调用.text会引发错误，大文件出现这类记录的概率更高。

解决方案（基于原代码修改） #

1. 修复文件读取编码 #

读取Endnote XML时指定UTF-8编码，避免乱码：

def convert_file(endnote_filepath: str='endnote_xml_file.xml', word_filepath: str='word_xml_file.xml'):
  with open(endnote_filepath, 'r', encoding='utf-8') as f:
    endnote_xml_str = f.read()
  save_word_file(endnote_xml_str, word_filepath)

2. 预处理XML字符串，转义特殊字符 #

添加预处理函数转义XML保留字符，避免解析报错：

def preprocess_xml(xml_str):
    # 注意转义顺序：&必须第一个处理，否则会把其他转义符的&也转义
    xml_str = xml_str.replace('&', '&')
    xml_str = xml_str.replace('<', '&lt;')
    xml_str = xml_str.replace('>', '&gt;')
    xml_str = xml_str.replace('"', '&quot;')
    xml_str = xml_str.replace("'", '&apos;')
    return xml_str

def convert_endnote_to_word(endnote_xml_str):
    # 先预处理XML字符串
    endnote_xml_str = preprocess_xml(endnote_xml_str)
    word_xml_str = '<?xml version="1.0" encoding="utf-8"?>\n'
    word_xml_str += '<b:Sources xmlns:b="http://schemas.openxmlformats.org/officeDocument/2006/bibliography" '
    word_xml_str += 'xmlns="http://schemas.openxmlformats.org/officeDocument/2006/bibliography" SelectedStyle="">\n' 
    root = ET.fromstring(endnote_xml_str)
    # 后续代码不变...

3. 改用ET.parse()替代fromstring() #

直接读取文件解析，避免字符串处理的潜在问题，更适合大文件：

# 修改convert_endnote_to_word接收文件路径而非字符串
def convert_endnote_to_word(endnote_filepath):
    word_xml_str = '<?xml version="1.0" encoding="utf-8"?>\n'
    word_xml_str += '<b:Sources xmlns:b="http://schemas.openxmlformats.org/officeDocument/2006/bibliography" '
    word_xml_str += 'xmlns="http://schemas.openxmlformats.org/officeDocument/2006/bibliography" SelectedStyle="">\n' 
    # 直接解析文件
    tree = ET.parse(endnote_filepath)
    root = tree.getroot()
    # 后续循环代码不变...

# 修改convert_file直接传递文件路径
def convert_file(endnote_filepath: str='endnote_xml_file.xml', word_filepath: str='word_xml_file.xml'):
    word_xml_str = convert_endnote_to_word(endnote_filepath)
    with open(word_filepath, 'w', encoding='utf-8') as f:
        f.write(word_xml_str)

# 同步修改save_word_file（如果不需要可以删除，直接在convert_file处理写入）
def save_word_file(word_xml_str, word_filepath: str='word_xml_file.xml'):
    with open(word_filepath, 'w', encoding='utf-8') as f:
      f.write(word_xml_str)

4. 处理字段可能为None的情况 #

避免因记录缺失字段导致程序崩溃：

for record in root.findall('./records/record'):
    source = {}
    word_xml_str += '  <b:Source>\n'
    # 处理year字段
    year_elem = record.find('./pub-dates/date')
    year = year_elem.text.strip() if (year_elem and year_elem.text) else ''
    source['year'] = year
    word_xml_str += f'    <b:Year>{source["year"]}</b:Year>\n'  
    # 处理title字段
    title_elem = record.find('./titles/title')
    title = title_elem.text.strip() if (title_elem and title_elem.text) else ''
    source['title'] = title
    word_xml_str += f'    <b:Title>{source["title"]}</b:Title>\n'
    # 处理url字段
    url_elem = record.find('./urls/related-urls/url')
    url = url_elem.text.strip() if (url_elem and url_elem.text) else ''
    source['url'] = url
    word_xml_str += f'    <b:URL>{source["url"]}</b:URL>\n'
    # 其他代码不变...

验证建议 #

1. 先测试包含特殊字符的单条记录，确认预处理和字段处理逻辑有效。
2. 逐步增加记录数量，验证大文件解析稳定性。

内容的提问来源于stack exchange，提问作者Fabricio Siqueira