Python爬虫提取文本出现\xad字符问题及代码优化咨询

一、软连字符\xad的处理方法 #

你遇到的\xad是Unicode软连字符（U+00AD），网页开发者用它实现长单词的自动换行，爬取时会被当作普通文本提取出来。解决方法很直接，把这个字符替换掉就行：

• 提取文本时实时处理：比如text.replace('\xad', '')
• 批量处理列表：用列表推导式遍历所有文本，统一替换

二、代码优化建议 #

1. 缩小异常捕获范围 #

原来的except:会捕获所有异常（包括语法错误这类不该被捕获的问题），应该只捕获请求和解析相关的具体异常：

from requests.exceptions import RequestException

def __init__(self, url):
    try:
        # 请求和解析代码
    except RequestException as e:
        raise ValueError(f"请求失败: {str(e)}")
    except etree.ParserError as e:
        raise ValueError(f"HTML解析失败: {str(e)}")

2. 优化编码处理 #

requests.get(url).text可能出现编码识别错误，改用response.content让lxml自动处理编码，或者先修正编码再取文本：

response = requests.get(url, headers=Page.headers)
response.encoding = response.apparent_encoding  # 用自动识别的编码修正
self.html = response.text

3. 改进XPath文本提取 #

直接用span.text可能拿到None，且只能获取节点的直接文本，改用xpath('string()')可以获取节点下所有文本，再配合清理函数处理软连字符：

# 提取时统一处理
text = node.xpath('string()').replace('\xad', '').strip()

4. 精简类成员 #

初始化里的self.var完全没用，直接删掉；self.articles可以在get_stories方法里填充，不用提前初始化空字典。

5. 拆分方法职责 #

把请求、文本清理、数据提取拆成单独方法，代码更易维护：

def _fetch_response(self, url):
    response = requests.get(url, headers=self.headers)
    response.raise_for_status()  # 触发HTTP错误异常
    response.encoding = response.apparent_encoding
    return response.text

def _clean_text(self, text):
    if not text:
        return ""
    return text.replace('\xad', '').strip()

6. 让方法返回数据而非直接打印 #

get_stories不要只做打印，改成返回整理好的数据，方便后续使用：

def get_stories(self):
    headers = [self._clean_text(node.xpath('string()')) 
               for node in self.tree.xpath('//a[@class="u-clickable-card__link"]//span')]
    snippets = [self._clean_text(node.xpath('string()')) 
                for node in self.tree.xpath('//div[@class="gc__excerpt"]//p')]
    self.articles = {"headers": headers, "snippets": snippets}
    return self.articles

修改后的完整代码 #

from lxml import etree
import requests
from requests.exceptions import RequestException

class Page:
    headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"}

    def __init__(self, url):
        try:
            html = self._fetch_response(url)
            self.tree = etree.HTML(html)
            self.articles = {}
        except (RequestException, etree.ParserError) as e:
            raise ValueError(f"页面初始化失败: {str(e)}")

    def _fetch_response(self, url):
        response = requests.get(url, headers=Page.headers)
        response.raise_for_status()
        response.encoding = response.apparent_encoding
        return response.text

    def _clean_text(self, text):
        if not text:
            return ""
        return text.replace('\xad', '').strip()

    def get_stories(self):
        header_nodes = self.tree.xpath('//a[@class="u-clickable-card__link"]//span')
        headers = [self._clean_text(node.xpath('string()')) for node in header_nodes]
        
        snippet_nodes = self.tree.xpath('//div[@class="gc__excerpt"]//p')
        snippets = [self._clean_text(node.xpath('string()')) for node in snippet_nodes]
        
        self.articles = {"headers": headers, "snippets": snippets}
        return self.articles

if __name__ == "__main__":
    url = "https://www.aljazeera.com/tag/julian-assange/"
    try:
        page1 = Page(url)
        stories = page1.get_stories()
        print(stories["headers"])
    except ValueError as e:
        print(e)

内容的提问来源于stack exchange，提问作者intmainvoid