使用BeautifulSoup提取指定表格td标签文本的问题求助
问题:使用BeautifulSoup提取表格td文本并转为DataFrame遇阻
背景
我正在用BeautifulSoup从一个AFL数据站点提取表格数据,该站点包含多个带唯一table id的表格。目前我已经通过以下代码成功定位到了目标表格:
from bs4 import BeautifulSoup from selenium import webdriver stat_dict = { 'Disposals': 'sortableTable0', 'Kicks': 'sortableTable1', 'Marks': 'sortableTable2', 'Handballs': 'sortableTable3', 'Goals': 'sortableTable4', 'Behinds': 'sortableTable5', 'Hitouts': 'sortableTable6', 'Tackles': 'sortableTable7', 'Rebounds': 'sortableTable8', 'Inside50s': 'sortableTable9', 'Clearances': 'sortableTable10', 'Clangers': 'sortableTable11', 'FreesFor': 'sortableTable12', 'FreesAgainst': 'sortableTable13', 'ContestedPosessions': 'sortableTable14', 'UncontestedPosesseions': 'sortableTable15', 'ContestedMarks': 'sortableTable16', 'MarksInside50': 'sortableTable17', 'OnePercenters': 'sortableTable18', 'Bounces': 'sortableTable19', 'GoalAssists': 'sortableTable20', 'Timeplayed': 'sortableTable21' } driver = webdriver.Firefox(executable_path='...') url = "https://afltables.com/afl/stats/teams/adelaide/2018_gbg.html" driver.get(url) html = driver.page_source soup = BeautifulSoup(html, "lxml") stat_wanted = 'Disposals' table = soup.find_all('table', {'id': stat_dict[stat_wanted]})
期望实现的功能
我希望实现和以下代码等效的功能——这段代码通过soup.find('tbody')可以正常运行,能把提取到的td文本整理成Pandas DataFrame:
import requests import pandas as pd import numpy as np from bs4 import BeautifulSoup def get_disposals(team_lower_case, nplayers, nrounds): list = [] page = requests.get("https://afltables.com/afl/stats/teams/" + str(team_lower_case) + "/2018_gbg.html") soup = BeautifulSoup(page.content, 'html.parser') filter = soup.find('tbody') for var in filter.find_all('tr'): columns = var.find_all('td') for val in columns: list.append(val.get_text()) columns = ['PlayerName'] for n in range(1, nrounds+1): columns.append('R'+str(n)) df = pd.DataFrame(np.array(list).reshape(nplayers, nrounds+1), columns=columns) return df get_disposals("fremantle",30,8)
当前遇到的问题
当我尝试用下面的代码提取所有td标签的文本时,没有得到预期的纯文本结果:
for tr in table: zxc = tr.find_all('td') print(zxc) for var in zxc: list = [] list.append(var.get_text()) print(list)
运行这段代码后,输出的是标签及其内容的列表,而不是get_text()应该返回的纯文本内容。我需要帮忙解决这个问题,让代码能正确提取纯文本并整理成DataFrame。
内容的提问来源于stack exchange,提问作者Kamila Ambro
相关产品推荐
相关产品推荐

