能否拆分Expedia爬取的航班数据并构建Pandas DataFrame？

我使用Selenium从Expedia网页爬取了如下航班数据：

flight_data = [
    'Select and show fare information for Eurowings flight, departing at 3:20pm from Helsinki, arriving at 4:30pm in Berlin, Priced at $53 One way per traveler.  2 hours 10 minutes total travel time, Nonstop.',
    'Select and show fare information for Air Baltic flight, departing at 5:20am from Helsinki, arriving at 7:00pm in Berlin, Priced at $84 One way per traveler, 3 left at this price.  14 hours 40 minutes total travel time, One stop, Layover for 12 hours 0 minutes in Riga.',
    'Select and show fare information for Air Baltic flight, departing at 9:20pm from Helsinki, arriving at 7:55am in Berlin, Priced at $94 One way per traveler. Arrives 1 day later. 11 hours 35 minutes total travel time, One stop, Layover for 8 hours 55 minutes in Riga.',
    'Select and show fare information for Scandinavian Airlines flight, departing at 10:05am from Helsinki, arriving at 3:15pm in Berlin, Priced at $142 One way per traveler.  6 hours 10 minutes total travel time, One stop, Layover for 3 hours 25 minutes in Copenhagen.• Scandinavian Airlines 1707 and 1673 operated by Sas Connect',
    'Select and show fare information for Lufthansa flight, departing at 7:30pm from Helsinki, arriving at 8:05am in Berlin, Priced at $150 One way per traveler. Arrives 1 day later. 13 hours 35 minutes total travel time, One stop, Layover for 9 hours 35 minutes in Munich.',
    'Select and show fare information for KLM flight, departing at 1:55pm from Helsinki, arriving at 5:40pm in Berlin, Priced at $163 One way per traveler.  4 hours 45 minutes total travel time, One stop, Layover for 0 hours 55 minutes in Amsterdam.',
    'Select and show fare information for KLM flight, departing at 7:00am from Helsinki, arriving at 11:05am in Berlin, Priced at $163 One way per traveler.  5 hours 5 minutes total travel time, One stop, Layover for 1 hour 0 minutes in Amsterdam.',
    'Select and show fare information for KLM flight, departing at 10:40am from Helsinki, arriving at 3:40pm in Berlin, Priced at $163 One way per traveler, 1 left at this price.  6 hours 0 minutes total travel time, One stop, Layover for 2 hours 0 minutes in Amsterdam.• KLM 1166 and 1829 operated by KLM Cityhopper',
    'Select and show fare information for KLM flight, departing at 10:40am from Helsinki, arriving at 5:40pm in Berlin, Priced at $163 One way per traveler.  8 hours 0 minutes total travel time, One stop, Layover for 4 hours 5 minutes in Amsterdam.• KLM 1166 operated by KLM Cityhopper',
    'Select and show fare information for KLM flight, departing at 1:55pm from Helsinki, arriving at 9:05pm in Berlin, Priced at $163 One way per traveler.  8 hours 10 minutes total travel time, One stop, Layover for 4 hours 2'
]

现计划拆分上述数据，构建包含指定字段的DataFrame，代码框架如下：

import pandas as pd

data_e = {
    "Airline": lst_airline_e,
    "Price": lst_prices_e,
    "Departure Time": departure_time_expedia,
    "Arrival Time": arrival_time_expedia,
    "Duration": lst_duration_e,
    "No of Stops": lst_stops_e,
    "Layover Time": lst_layover_e
}
df_m = pd.DataFrame.from_dict(data_e, orient='index')
df_m

请问是否可以实现该数据拆分操作？

完全可以实现。可以通过正则表达式匹配每条数据中的固定格式字段，提取所需信息后填充到对应列表，最终构建DataFrame。以下是具体实现代码：

import pandas as pd
import re

# 爬取到的航班数据
flight_data = [
    'Select and show fare information for Eurowings flight, departing at 3:20pm from Helsinki, arriving at 4:30pm in Berlin, Priced at $53 One way per traveler.  2 hours 10 minutes total travel time, Nonstop.',
    'Select and show fare information for Air Baltic flight, departing at 5:20am from Helsinki, arriving at 7:00pm in Berlin, Priced at $84 One way per traveler, 3 left at this price.  14 hours 40 minutes total travel time, One stop, Layover for 12 hours 0 minutes in Riga.',
    'Select and show fare information for Air Baltic flight, departing at 9:20pm from Helsinki, arriving at 7:55am in Berlin, Priced at $94 One way per traveler. Arrives 1 day later. 11 hours 35 minutes total travel time, One stop, Layover for 8 hours 55 minutes in Riga.',
    'Select and show fare information for Scandinavian Airlines flight, departing at 10:05am from Helsinki, arriving at 3:15pm in Berlin, Priced at $142 One way per traveler.  6 hours 10 minutes total travel time, One stop, Layover for 3 hours 25 minutes in Copenhagen.• Scandinavian Airlines 1707 and 1673 operated by Sas Connect',
    'Select and show fare information for Lufthansa flight, departing at 7:30pm from Helsinki, arriving at 8:05am in Berlin, Priced at $150 One way per traveler. Arrives 1 day later. 13 hours 35 minutes total travel time, One stop, Layover for 9 hours 35 minutes in Munich.',
    'Select and show fare information for KLM flight, departing at 1:55pm from Helsinki, arriving at 5:40pm in Berlin, Priced at $163 One way per traveler.  4 hours 45 minutes total travel time, One stop, Layover for 0 hours 55 minutes in Amsterdam.',
    'Select and show fare information for KLM flight, departing at 7:00am from Helsinki, arriving at 11:05am in Berlin, Priced at $163 One way per traveler.  5 hours 5 minutes total travel time, One stop, Layover for 1 hour 0 minutes in Amsterdam.',
    'Select and show fare information for KLM flight, departing at 10:40am from Helsinki, arriving at 3:40pm in Berlin, Priced at $163 One way per traveler, 1 left at this price.  6 hours 0 minutes total travel time, One stop, Layover for 2 hours 0 minutes in Amsterdam.• KLM 1166 and 1829 operated by KLM Cityhopper',
    'Select and show fare information for KLM flight, departing at 10:40am from Helsinki, arriving at 5:40pm in Berlin, Priced at $163 One way per traveler.  8 hours 0 minutes total travel time, One stop, Layover for 4 hours 5 minutes in Amsterdam.• KLM 1166 operated by KLM Cityhopper',
    'Select and show fare information for KLM flight, departing at 1:55pm from Helsinki, arriving at 9:05pm in Berlin, Priced at $163 One way per traveler.  8 hours 10 minutes total travel time, One stop, Layover for 4 hours 2'
]

# 初始化存储各字段的列表
lst_airline_e = []
lst_prices_e = []
departure_time_expedia = []
arrival_time_expedia = []
lst_duration_e = []
lst_stops_e = []
lst_layover_e = []

# 正则匹配模板，覆盖所有字段的提取规则
pattern = re.compile(
    r'for (\w+(?: \w+)?) flight, departing at (\d+:\d+[ap]m) from .*, arriving at (\d+:\d+[ap]m) in .*, Priced at \$(\d+) .*? (\d+ hours? \d+ minutes?) total travel time, (Nonstop|One stop)(?:, Layover for (\d+ hours? \d+ minutes?))?'
)

for entry in flight_data:
    # 处理最后一条不完整的数据
    if entry.endswith('4 hours 2'):
        entry += '0 minutes in Amsterdam.'
    
    match = pattern.search(entry)
    if match:
        airline, dep_time, arr_time, price, duration, stops, layover = match.groups()
        
        lst_airline_e.append(airline)
        lst_prices_e.append(int(price))
        departure_time_expedia.append(dep_time)
        arrival_time_expedia.append(arr_time)
        lst_duration_e.append(duration)
        
        # 转换转机次数为数字
        lst_stops_e.append(0 if stops == 'Nonstop' else 1)
        
        # 无转机时 layover 为 None，设为空字符串
        lst_layover_e.append(layover if layover else '')

# 构建DataFrame（原代码的orient='index'会把字段作为行，这里改为常规列模式，若需保持原格式可添加.T转置）
data_e = {
    "Airline": lst_airline_e,
    "Price": lst_prices_e,
    "Departure Time": departure_time_expedia,
    "Arrival Time": arrival_time_expedia,
    "Duration": lst_duration_e,
    "No of Stops": lst_stops_e,
    "Layover Time": lst_layover_e
}
df_m = pd.DataFrame(data_e)
print(df_m)

说明 #

1. 正则表达式匹配了每条数据中的航空公司、起降时间、价格、总时长、转机情况及中转时长，覆盖了所有有效数据格式。
2. 针对最后一条不完整的爬取数据，做了补全处理，避免匹配失败。
3. 原代码中的orient='index'会将字段作为行索引，改为默认列模式更符合常规DataFrame结构，若需保持原格式，可改为df_m = pd.DataFrame.from_dict(data_e, orient='index').T完成转置。

内容的提问来源于stack exchange，提问作者Gayani Thakshila