EDGAR财报文本中嵌入图形标签的高效去除方案问询

Great question—dealing with those bloated uuencoded image blocks in EDGAR 10-Ks is a huge hassle when processing these documents at scale. Your initial thought to strip content between begin 644 and end is a good starting point, but there are more robust, efficient approaches tailored to different use cases:

1. Robust Regular Expression Matching #

Your basic regex idea works, but we can make it more flexible to handle edge cases:

• EDGAR filings might use different permission codes (not just 644, though 644 is most common)
• Some blocks might have uppercase BEGIN/END
• Filenames for images are often included right after the permission code, which we can use to target only image-related blocks

Here’s a refined regex pattern (Python example):

import re

def clean_edgar_images(text):
    # Matches uuencoded image blocks: starts with begin [3 digits], includes image extension, ends with end
    pattern = r'begin \d{3}.*?\.(gif|jpg|jpeg|png).*?end'
    return re.sub(pattern, '', text, flags=re.DOTALL | re.IGNORECASE)

• re.DOTALL lets the .*? match across multiple lines
• re.IGNORECASE handles uppercase/lowercase variations of begin/end
• Adding the image extension check reduces the risk of accidentally removing non-image uuencoded content (though rare in 10-Ks)

2. Stream Processing for Large Files #

If you’re working with massive 10-K filings that can’t fit easily into memory, a line-by-line streaming approach is more efficient:

def clean_large_edgar_file(input_path, output_path):
    in_image_block = False
    with open(input_path, 'r', encoding='utf-8') as infile, open(output_path, 'w', encoding='utf-8') as outfile:
        for line in infile:
            # Check if we're entering an image block
            if re.match(r'begin \d{3}', line, re.IGNORECASE):
                in_image_block = True
                continue
            # Check if we're exiting an image block
            if re.match(r'end', line, re.IGNORECASE):
                in_image_block = False
                continue
            # Only write lines if we're not inside an image block
            if not in_image_block:
                outfile.write(line)

This method uses constant memory regardless of file size, making it ideal for batch processing hundreds of filings.

3. Batch Processing Script #

To scale this across multiple files, wrap the cleaning logic in a script that iterates over a directory:

import os

def batch_clean_edgar_files(input_dir, output_dir):
    os.makedirs(output_dir, exist_ok=True)
    for filename in os.listdir(input_dir):
        if filename.endswith('.txt'):  # Adjust extension to match your files
            input_path = os.path.join(input_dir, filename)
            output_path = os.path.join(output_dir, f'cleaned_{filename}')
            clean_large_edgar_file(input_path, output_path)
            print(f"Cleaned: {filename}")

# Usage
batch_clean_edgar_files('./raw_10ks', './cleaned_10ks')

Key Notes #

• Always spot-check a few files after cleaning to ensure no legitimate text was removed (though the image extension filter minimizes this risk)
• If you’re working with HTML-formatted 10-Ks, you could combine this with an HTML parser (like BeautifulSoup) to target only blocks within <pre> or other code-like tags, though this adds complexity for minimal gain in most cases

内容的提问来源于stack exchange，提问作者user21022587