爬取天气数据时如何将Pandas单元格字典转为DataFrame列

你要实现的效果可以用pandas内置的pd.json_normalize()一步完成，这是目前处理字典转DataFrame最简洁的Pythonic写法，不需要手动遍历拆包。

修改后的完整代码 #

import requests, re, typing
from bs4 import BeautifulSoup as soup
import contextlib
import pandas as pd

def _remove(d:list) -> list:
    return list(filter(None, [re.sub('\xa0', '', b) for b in d]))

@contextlib.contextmanager
def get_weather_data(url:str, by_url = True) -> typing.Generator[dict, None, None]:
    d = soup(requests.get(url).text if by_url else url, 'html.parser')
    _table = d.find('table', {'id':'wt-his'})
    _data = [[[i.text for i in c.find_all('th')], *[i.text for i in c.find_all('td')]] for c in _table.find_all('tr')]
    [h1], [h2], *data, _ = _data
    _h2 = _remove(h2)
    yield {tuple(_remove(h1)):[dict(zip(_h2, _remove([a, *i]))) for [[a], *i] in data]}

city = 'https://www.timeanddate.com/weather/germany/berlin/historic?hd=20210905'
with get_weather_data(city) as weather:
    # 直接取出所有时间点的字典列表
    weather_list = next(iter(weather.values()))
    # 一行转换为DataFrame
    df = pd.json_normalize(weather_list)
    # 按需筛选需要的列，可根据实际返回的字段名调整
    df = df[['Time', 'Temp', 'Wind', 'Humidity']]
    # 可选：清洗字段，提取数值去除单位，方便后续分析
    df['Temp'] = df['Temp'].str.extract('(\d+)').astype(int)
    df['Wind'] = df['Wind'].str.extract('(\d+)').astype(int)
    df['Humidity'] = df['Humidity'].str.extract('(\d+)').astype(int)
    print(df.head())

说明 #

• pd.json_normalize()会自动识别字典的所有键作为DataFrame的列名，每个键对应的值作为行数据，完美适配你当前每个时间戳对应一个字典的结构
• 后续清洗步骤可以根据你的实际需求调整，如果不需要转换数值类型可以直接删除相关代码

内容的提问来源于stack exchange，提问作者Medulla Oblongata