如何基于爬取的酒店数据创建含名称与评论列的DataFrame

我用以下Python代码爬取Booking.com的酒店数据，但目前获取到的酒店名称和评论数据混存在同一个列表中，希望将这些数据整理为带有「名称」和「评论」独立列的DataFrame，预期效果如图所示：

原爬取代码 #

import requests, re
from bs4 import BeautifulSoup

data = []

soup = BeautifulSoup(
    requests.get('https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4ARfIAQzYAQHoAQH4AQKIAgGoAgO4AuS4sJ4GwAIB0gIkYWJlYmZiMWItNWJjMi00M2Y2LTk3MGUtMzI2ZGZmMmIyNzMz2AIF4AIB&aid=304142&dest_id=-2092174&dest_type=city&group_adults=2&req_adults=2&no_rooms=1&group_children=0&req_children=0&nflt=ht_id%3D204&rows=15',
                headers={'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
                ).text)

num_results = int(re.search(r'\d+',soup.select_one('div:has(+[data-testid="pagination"])').text).group(0))

for i in range(0,int(num_results/25)):
    soup = BeautifulSoup(
    requests.get(f'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4ARfIAQzYAQHoAQH4AQKIAgGoAgO4AuS4sJ4GwAIB0gIkYWJlYmZiMWItNWJjMi00M2Y2LTk3MGUtMzI2ZGZmMmIyNzMz2AIF4AIB&aid=304142&dest_id=-2092174&dest_type=city&group_adults=2&req_adults=2&no_rooms=1&group_children=0&req_children=0&nflt=ht_id%3D204&rows=15&offset={int(i*25)}',
                headers={'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
            ).text
    )
    data.extend([e.select_one('[data-testid="title"]').text for e in soup.select('[data-testid="property-card"]')])
    data.extend([e.select_one('[class="d8eab2cf7f c90c0a70d3 db63693c62"]') for e in soup.select('[data-testid="property-card"]')])

data

问题分析 #

原代码的问题在于：每次循环先将所有酒店名称一次性追加到列表，再追加所有评论数据，导致列表结构为[名称1, 名称2, ..., 评论1, 评论2, ...]，两类数据完全分离，无法一一对应。

解决方案 #

修改爬取逻辑，逐个处理每个酒店卡片，同时获取当前酒店的名称和评论数据，将它们作为一组存入列表，最后直接转换为DataFrame。具体步骤：

1. 导入pandas库用于生成DataFrame
2. 遍历每个酒店卡片时，同时提取名称和评论（处理评论为空的情况）
3. 将每组数据以字典形式存入列表
4. 用列表生成包含「名称」「评论」列的DataFrame

修改后的完整代码 #

import requests, re
import pandas as pd
from bs4 import BeautifulSoup

# 初始化存储字典的列表
hotel_data = []

# 初始请求获取总结果数
soup = BeautifulSoup(
    requests.get(
        'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4ARfIAQzYAQHoAQH4AQKIAgGoAgO4AuS4sJ4GwAIB0gIkYWJlYmZiMWItNWJjMi00M2Y2LTk3MGUtMzI2ZGZmMmIyNzMz2AIF4AIB&aid=304142&dest_id=-2092174&dest_type=city&group_adults=2&req_adults=2&no_rooms=1&group_children=0&req_children=0&nflt=ht_id%3D204&rows=15',
        headers={'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
    ).text,
    'html.parser'
)

num_results = int(re.search(r'\d+', soup.select_one('div:has(+[data-testid="pagination"])').text).group(0))

# 遍历分页
for i in range(0, int(num_results/25)):
    soup = BeautifulSoup(
        requests.get(
            f'https://www.booking.com/searchresults.html?label=gen173nr-1FCAEoggI46AdIM1gEaGyIAQGYATG4ARfIAQzYAQHoAQH4AQKIAgGoAgO4AuS4sJ4GwAIB0gIkYWJlYmZiMWItNWJjMi00M2Y2LTk3MGUtMzI2ZGZmMmIyNzMz2AIF4AIB&aid=304142&dest_id=-2092174&dest_type=city&group_adults=2&req_adults=2&no_rooms=1&group_children=0&req_children=0&nflt=ht_id%3D204&rows=15&offset={int(i*25)}',
            headers={'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
        ).text,
        'html.parser'
    )
    
    # 逐个处理酒店卡片
    for card in soup.select('[data-testid="property-card"]'):
        # 获取酒店名称
        name = card.select_one('[data-testid="title"]').text.strip()
        # 获取评论分数，处理可能为空的情况
        rating_elem = card.select_one('[class="d8eab2cf7f c90c0a70d3 db63693c62"]')
        rating = rating_elem.text.strip() if rating_elem else None
        
        # 将当前酒店数据加入列表
        hotel_data.append({'名称': name, '评论': rating})

# 转换为DataFrame
df = pd.DataFrame(hotel_data)
print(df)

说明 #

• 修改了user-agent为真实浏览器标识，降低被网站拦截的概率
• 加入了评论为空的处理逻辑，避免代码因元素缺失报错
• 最终生成的DataFrame会直接呈现「名称」和「评论」两列，与预期效果一致

内容的提问来源于stack exchange，提问作者Sasksham mishra