如何将亚马逊图书信息提取并转换为表格形式？

我已经通过Selenium加载亚马逊商品页面，并用BeautifulSoup提取了商品详情的HTML列表，现有代码如下：

from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

link = 'https://www.amazon.com/Functional-Analysis-Dzung-Ha/dp/0971576610'
driver = webdriver.Chrome(service = Service(ChromeDriverManager().install()))
driver.get(link)
soup = BeautifulSoup(driver.page_source, 'html.parser')
details_list = soup.select("#detailBullets_feature_div .a-unordered-list:first-child")[0]

提取到的HTML结构为：

<ul class="a-unordered-list a-nostyle a-vertical a-spacing-none detail-bullet-list">
<li><span class="a-list-item"> <span class="a-text-bold">Publisher : </span> <span>Matrix Editions (January 1, 2007)</span> </span></li>
<li><span class="a-list-item"> <span class="a-text-bold">Language : </span> <span>English</span> </span></li>
<li><span class="a-list-item"> <span class="a-text-bold">Hardcover : </span> <span>640 pages</span> </span></li>
<li><span class="a-list-item"> <span class="a-text-bold">ISBN-10 : </span> <span>0971576610</span> </span></li>
<li><span class="a-list-item"> <span class="a-text-bold">ISBN-13 : </span> <span>978-0971576612</span> </span></li>
<li><span class="a-list-item"> <span class="a-text-bold">Item Weight : </span> <span>2.3 pounds</span> </span></li>
</ul>

转换方法 #

要把上述HTML转换成表格，只需遍历列表项提取键值对，再用Pandas生成表格即可，具体步骤如下：

1. 提取键值对

遍历每个<li>标签，分别提取加粗的属性名和对应的属性值：

data = []
for li in details_list.find_all('li'):
    # 提取属性名并去掉末尾的冒号
    attr_name = li.find('span', class_='a-text-bold').text.strip().replace(':', '')
    # 提取属性值
    attr_value = li.find('span', class_=False).text.strip()
    data.append({'属性': attr_name, '内容': attr_value})

2. 生成表格

用Pandas将提取到的键值对转换成格式化表格：

df = pd.DataFrame(data)
print(df.to_markdown(index=False))

最终效果 #

执行代码后会输出如下表格：

完整代码 #

from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

link = 'https://www.amazon.com/Functional-Analysis-Dzung-Ha/dp/0971576610'
driver = webdriver.Chrome(service = Service(ChromeDriverManager().install()))
driver.get(link)
soup = BeautifulSoup(driver.page_source, 'html.parser')
details_list = soup.select("#detailBullets_feature_div .a-unordered-list:first-child")[0]

# 提取键值对
data = []
for li in details_list.find_all('li'):
    attr_name = li.find('span', class_='a-text-bold').text.strip().replace(':', '')
    attr_value = li.find('span', class_=False).text.strip()
    data.append({'属性': attr_name, '内容': attr_value})

# 生成并输出表格
df = pd.DataFrame(data)
print(df.to_markdown(index=False))

driver.quit()

内容的提问来源于stack exchange，提问作者Akira