如何用lxml在XPath定位元素中控制子节点及解决text为空问题

核心原因 #

element.text仅返回当前元素的直接子文本节点，若目标文本嵌套在子标签中，就会返回None或空值。

可行解决方案 #

1. 使用XPath的string()或text()函数 #

针对单个元素，可直接通过XPath表达式获取完整文本：

# 示例代码修改（Nike维基页面）
from bs4 import BeautifulSoup
from lxml import etree, html
import requests
URL = "https://en.wikipedia.org/wiki/Nike,_Inc."

HEADERS = ({'User-Agent':
            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \
            (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',\
            'Accept-Language': 'en-US, en;q=0.5'})

webpage = requests.get(URL, headers=HEADERS)
soup = BeautifulSoup(webpage.content, "html.parser")
dom = etree.HTML(str(soup))

# 方法1：获取直接子文本节点
print(dom.xpath('//*[@id="firstHeading"]/text()')[0])
# 方法2：获取元素下所有层级的合并文本
print(dom.xpath('string(//*[@id="firstHeading"])'))

2. 循环处理元素时的文本提取 #

针对测试代码中的循环场景，用以下方式获取文本：

import pytest
from bs4 import BeautifulSoup
from lxml import etree, html
import requests

def test_scraping():
    URL = "https://news.yahoo.co.jp/search?p=岸田文雄&ei=utf-8&categories=business"

    HEADERS = ({'User-Agent':
                'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \
                (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',\
                'Accept-Language': 'en-US, en;q=0.5'})

    webpage = requests.get(URL, headers=HEADERS)
    soup = BeautifulSoup(webpage.content, "html.parser")
    dom = etree.HTML(str(soup))
    elements = dom.xpath("//li[@class='viewableWrap newsFeed_item newsFeed_item-normal newsFeed_item-ranking']")
    
    for element in elements:
        print(element.tag)
        
        # 方式1：获取元素下所有合并文本（自动去除多余空白）
        full_text = element.xpath('string()').strip()
        if full_text:
            print(full_text)
        
        # 方式2：获取所有子节点文本列表，按需清洗拼接
        text_list = element.xpath('.//text()')
        cleaned_texts = [t.strip() for t in text_list if t.strip()]
        print(' | '.join(cleaned_texts))

3. 简化流程：直接用lxml解析响应内容 #

无需同时使用BeautifulSoup和lxml，直接用lxml解析请求响应，减少转换损耗：

webpage = requests.get(URL, headers=HEADERS)
dom = etree.HTML(webpage.content)
# 后续直接用xpath操作即可

内容的提问来源于stack exchange，提问作者Jun Takeshita