如何将BeautifulSoup爬取输出转换为字典以生成Pandas DataFrame

你可以按以下方案修改代码，实现将爬取内容转换为字典并生成Pandas DataFrame的需求：

核心思路 #

你通过正则拿到的属性列表遵循类别名 + 对应属性序列 + 下一个类别名 + 对应属性序列的排列规律，只需要遍历列表归类属性，再收集所有宝可梦的记录即可直接生成DataFrame。

修改后的完整代码 #

import time
import re
import pandas as pd
from bs4 import BeautifulSoup

# 预定义属性克制类别
CATEGORIES = {'Immune to:', 'Strongly resists:', 'Resists:', 'Weak to:', 'Very weak to:'}
# 存储所有宝可梦数据的列表
all_pokemon_data = []

pokemons = driver.find_elements_by_xpath("//div[@class='PokemonAltRow-name']")
for count, pokemon in enumerate(pokemons):
    pokemons = driver.find_elements_by_xpath("//div[@class='PokemonAltRow-name']")
    driver.implicitly_wait(1)
    pokemons[count].click()
    time.sleep(1)
    page_source = driver.page_source
    time.sleep(1)
    soup = BeautifulSoup(page_source, 'html.parser')
    time.sleep(1)
    name = soup.find('h1', attrs={'data-reactid' : '.0.1.1.1'})
    type_effectives = soup.find('dl', attrs={'class' : 'TypeEffectives'})
    type_effectives_result = re.findall('[A-Z][^A-Z]*', type_effectives.text)

    # 处理当前宝可梦的属性归类
    current_pokemon = {
        '宝可梦名称': name.text,
        'Immune to:': [],
        'Strongly resists:': [],
        'Resists:': [],
        'Weak to:': [],
        'Very weak to:': []
    }
    current_category = None
    for item in type_effectives_result:
        if item in CATEGORIES:
            current_category = item
        else:
            if current_category:
                current_pokemon[current_category].append(item)
    all_pokemon_data.append(current_pokemon)

    driver.back()
    driver.implicitly_wait(1)

driver.close()

# 生成DataFrame
df = pd.DataFrame(all_pokemon_data)
# 可选：将列表格式的属性转为逗号分隔的字符串，方便导出存储
df[list(CATEGORIES)] = df[list(CATEGORIES)].applymap(lambda x: ','.join(x))
# 可选：导出为csv文件
df.to_csv('宝可梦属性克制表.csv', index=False, encoding='utf_8_sig')

输出说明 #

生成的DataFrame每行对应一只宝可梦，每列分别对应宝可梦名称、免疫属性、强抵抗属性、抵抗属性、弱属性、极弱属性，可直接用于后续分析。

内容的提问来源于stack exchange，提问作者Stefan van den Berg