如何将Beautiful Soup结果转为列表？网页解析文本异常问题咨询

Hey there! Let's sort out your parsing issues and get that table data into a clean list, step by step.

First, why are you seeing messy HTML entities and unstructured text? #

Your current approach uses soup.get_text() which yanks all text from the entire page, ignoring the HTML structure. Plus, the default html.parser built into Python can struggle with poorly formatted HTML (like the unclosed <a> tags in your example), leaving behind unprocessed entities like < instead of converting them to <.

Fix 1: Use a better parser for messy HTML #

BeautifulSoup works best with parsers like lxml or html5lib (instead of the default html.parser) when dealing with non-standard HTML. These parsers do a better job of fixing unclosed tags and decoding entities automatically.

First, install one if you haven't:

pip install html5lib  # or lxml

Fix 2: Extract structured table data (don't grab all text at once) #

Instead of pulling the entire page's text, target the <table> element directly, then iterate through rows and cells to build your list properly. Here's a revised code example:

# Replace urllib2 with requests (it's more modern and easier to use)
import requests
from bs4 import BeautifulSoup

url = "https://somewebpage.com"
response = requests.get(url)

# Use html5lib to handle messy HTML
soup = BeautifulSoup(response.content, "html5lib")

# Locate your target table (adjust if there are multiple tables: use find_all or add class/id selectors)
target_table = soup.find("table")

table_data = []
# Iterate through each row in the table
for row in target_table.find_all("tr"):
    # Get all cells in the row
    cells = row.find_all("td")
    # Process each cell to clean up text and handle entities
    cleaned_row = []
    for cell in cells:
        # Extract clean text (strip removes extra spaces/newlines; BeautifulSoup decodes entities automatically)
        cell_text = cell.get_text(strip=True)
        
        # If you need to extract attributes (like the progress bar's title), add logic here:
        progress_bar = cell.find("div", class_="ui-progressbar")
        if progress_bar:
            # Grab the title attribute instead of raw text
            cleaned_row.append(progress_bar.get("title", ""))
        else:
            cleaned_row.append(cell_text)
    
    # Skip empty rows (common in some tables)
    if cleaned_row:
        table_data.append(cleaned_row)

# Now you have a clean list of table rows!
print(table_data)

Key improvements here: #

• Structured extraction: By targeting <table>, <tr>, and <td> tags, you preserve the table's row/column structure instead of getting a jumble of text.
• Entity handling: html5lib and BeautifulSoup's get_text() automatically decode entities like < to their proper characters.
• Flexibility: Added logic to extract attributes (like the progress bar's title) if needed, since your example included progress bar elements.

If you still need to use urllib2 (instead of requests) #

Here's a quick adaptation for urllib2 users:

import urllib2
from bs4 import BeautifulSoup

page = urllib2.urlopen("https://somewebpage.com")
soup = BeautifulSoup(page, "html5lib")  # Switch parser here
# Rest of the table extraction code is identical to the example above

内容的提问来源于stack exchange，提问作者chowpay