网页爬取英国航空评论后，如何将列表数据转为结构化DataFrame

我正在爬取英国航空的评论页面，已经获取了乘客评论和评分数据，但不知道怎么把现有的列表转换成目标结构化DataFrame。

原爬取代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

base_url = "https://www.airlinequality.com/airline-reviews/british-airways"
pages = 5 #10
page_size = 1 #100

reviews = []
aircraft = []
seat_type = []
route = []
recommended = []
rating = []
category = []

for i in range(1, pages + 1):

    print(f"Scraping page {i}")

    # Create URL to collect links from paginated data
    url = f"{base_url}/page/{i}/?sortby=post_date%3ADesc&pagesize={page_size}"

    # Collect HTML data from this page
    response = requests.get(url)

    # Parse content
    content = response.content
    parsed_content = BeautifulSoup(content, 'html.parser')
    for para in parsed_content.find_all("div", {"class": "text_content"}):
        reviews.append(para.get_text())
        
    for para2 in parsed_content.find_all("div", {"class" : "review-stats"}):
        for para3 in para2.find_all('td',{'class' : 'review-value'}):
            rating.append(para3.get_text())
        recomend = rating[-1]
        rating = rating[:-1]
        for para4 in para2.find_all('td',{'class' : 'review-rating-stars stars'}):
            para5 = len(para4.find_all('span', {'class' : 'star fill'}))
            rating.append(para5)
        rating.append(recomend)
        #print(rating)
        for para6 in para2.find_all('td',{'class' : 'review-rating-header'}):
            category.append(para6.get_text())
        #print(category)
        
    print(f"   ---> {len(reviews)} total reviews")

当前问题 #

现在category和rating是多次循环追加后的扁平列表：

• 第一次循环：category = [a, b, c, d, e]，rating = [1, 2, 3, 4, 5]
• 第二次循环：category追加[a, c, e, o, p, q]，rating追加[9, 8, 7, 6, 5, 4]
• 最终得到：category = [a, b, c, d, e, a, c, e, o, p, q]，rating = [1, 2, 3, 4, 5, 9, 8, 7, 6, 5, 4]

我需要的是每个评论对应一行，各评分类别作为列，对应评分作为列值的结构化DataFrame。

解决方案 #

调整爬取逻辑，每次处理单个评论的评分时生成独立字典，避免全局列表的混乱，最后直接转换为DataFrame：

import requests
from bs4 import BeautifulSoup
import pandas as pd

base_url = "https://www.airlinequality.com/airline-reviews/british-airways"
pages = 5  # 可修改为目标页数
page_size = 1  # 可修改为每页爬取数量

reviews = []
ratings_list = []  # 存储每个评论的评分字典

for i in range(1, pages + 1):
    print(f"Scraping page {i}")

    url = f"{base_url}/page/{i}/?sortby=post_date%3ADesc&pagesize={page_size}"
    response = requests.get(url)
    parsed_content = BeautifulSoup(response.content, 'html.parser')

    # 爬取评论内容
    for para in parsed_content.find_all("div", {"class": "text_content"}):
        reviews.append(para.get_text())

    # 爬取单条评论的所有评分信息
    for review_stats in parsed_content.find_all("div", {"class": "review-stats"}):
        single_rating = {}
        # 处理文本类评分（如Seat Type、Route等）
        for value_td in review_stats.find_all('td', {'class': 'review-value'}):
            header = value_td.find_previous_sibling('td', {'class': 'review-rating-header'}).get_text()
            single_rating[header] = value_td.get_text()
        # 处理星级评分（如Inflight Entertainment等）
        for star_td in review_stats.find_all('td', {'class': 'review-rating-stars stars'}):
            header = star_td.find_previous_sibling('td', {'class': 'review-rating-header'}).get_text()
            star_count = len(star_td.find_all('span', {'class': 'star fill'}))
            single_rating[header] = star_count
        # 将当前评论的评分字典加入列表
        ratings_list.append(single_rating)

    print(f"   ---> {len(reviews)} total reviews")

# 合并评论与评分数据生成DataFrame
df = pd.DataFrame(ratings_list)
df['Review'] = reviews

# 调整列顺序，将评论放在第一列
cols = ['Review'] + [col for col in df.columns if col != 'Review']
df = df[cols]

print(df.head())

修改说明 #

1. 用ratings_list存储每个评论的独立评分字典，键为评分类别，值为对应评分
2. 通过find_previous_sibling直接关联评分与对应类别，避免全局列表的拼接混乱
3. 最后将评论内容作为新列加入DataFrame，缺失的评分类别会自动填充NaN，完全符合结构化需求

内容的提问来源于stack exchange，提问作者Raffel Ravionaldo