BeautifulSoup的':-soup-contains'伪类未实现问题求助

问题根源 #

1. 解析器不支持：:-soup-contains是BeautifulSoup扩展的CSS伪类，仅兼容lxml或html5lib解析器，默认的html.parser不提供该功能。
2. 转义字符错误：代码中使用HTML转义字符"，导致选择器无法正确识别目标文本。
3. requests调用参数错误：requests.get(url,'html.parser')的第二个参数传递错误，解析器参数应传给BeautifulSoup，而非requests。

修复方案 #

• 安装lxml解析器（推荐，性能更优）：pip install lxml
• 将所有"替换为双引号"
• 修正requests与BeautifulSoup的调用逻辑，指定正确解析器
• 优化选择器逻辑，改用更稳定的元素匹配方式

修改后的完整代码 #

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
import unicodedata

cities = ['Berlin','Paris','Amsterdam','Barcelona','Rome','Lisbon','Prague','Vienna','Madrid']

def City_info(soup):
    ret_dict = {}
    ret_dict['city'] = soup.h1.get_text()
    
    # 获取市长信息
    mayor_row = soup.select_one('.mergedrow:-soup-contains("Mayor")')
    if mayor_row:
        mayor_name_html = mayor_row.find(class_='infobox-data')
        if mayor_name_html:
            ret_dict['mayor'] = unicodedata.normalize('NFKD', mayor_name_html.get_text(strip=True))
    
    # 获取城市面积
    area_row = soup.select_one('.mergedrow:-soup-contains("City")')
    if area_row:
        area_data = area_row.find(class_='infobox-data')
        if area_data:
            ret_dict['city_size'] = unicodedata.normalize('NFKD', area_data.get_text(strip=True))
    
    # 获取海拔
    elevation_row = soup.select_one('.mergedtoprow:-soup-contains("Elevation")')
    if elevation_row:
        elevation_data = elevation_row.find(class_='infobox-data')
        if elevation_data:
            ret_dict['elevation'] = unicodedata.normalize('NFKD', elevation_data.get_text(strip=True))
    
    # 获取城市人口
    population_row = soup.select_one('.mergedtoprow:-soup-contains("Population")')
    if population_row:
        pop_data = population_row.find_next('td')
        if pop_data:
            ret_dict['city_population'] = unicodedata.normalize('NFKD', pop_data.get_text(strip=True))
    
    # 获取都市圈人口
    urban_pop_label = soup.select_one('.infobox-label[title^=Urban]')
    if urban_pop_label:
        urban_pop_data = urban_pop_label.find_next('td')
        if urban_pop_data:
            ret_dict['urban_population'] = unicodedata.normalize('NFKD', urban_pop_data.get_text(strip=True))
    
    # 获取大都会区人口
    metro_pop_label = soup.select_one('.infobox-label[title^=Metro]')
    if metro_pop_label:
        metro_pop_data = metro_pop_label.find_next('td')
        if metro_pop_data:
            ret_dict['metro_population'] = unicodedata.normalize('NFKD', metro_pop_data.get_text(strip=True))
    
    # 获取经纬度
    lat = soup.select_one('.latitude')
    if lat:
        ret_dict['lat'] = lat.get_text(strip=True)
    long_ = soup.select_one('.longitude')
    if long_:
        ret_dict['long'] = long_.get_text(strip=True)
    
    return ret_dict

list_of_city_info = []
for city in cities:
    url = f'https://en.wikipedia.org/wiki/{city}'
    response = requests.get(url)
    # 使用lxml解析器
    soup = bs(response.content, 'lxml')
    list_of_city_info.append(City_info(soup))

df_cities = pd.DataFrame(list_of_city_info).set_index('city')
print(df_cities)

额外说明 #

若不想安装lxml，也可使用html5lib解析器（安装命令：pip install html5lib），只需将BeautifulSoup初始化代码改为：

soup = bs(response.content, 'html5lib')

内容的提问来源于stack exchange，提问作者Gurpreet Singh