网页爬取：修正li标签数据提取错误及多页批量爬取实现

一、修正字段与DataFrame列的对应问题 #

你的代码错误在于按顺序填充所有<li>文本，但页面里的<li>是字段名+对应值成对出现的（比如第一个是"DRAMP ID"，第二个是"DRAMP00005"），需要拆分成对内容后匹配DataFrame列。

修改后的详情页提取代码：

from bs4 import BeautifulSoup
import requests as rq
import pandas as pd

base_url = 'http://dramp.cpu-bioinfor.org/browse/All_Information.php?id='
url = f'{base_url}DRAMP00005'
page = rq.get(url)
htmlSoup = BeautifulSoup(page.content, "lxml")

# 定义目标列
cols = ['DRAMP ID','Peptide Name','Source','Family','Gene', 'Sequence','Sequence Length','UniProt Entry','Protein Existence', 'Biological Activity','Target Organism','Hemolytic Activity','Cytotoxicity','Binding Target']
new_table = pd.DataFrame(columns=cols)

# 处理General Information板块
general_section = htmlSoup.select("div.bs-docs-section")[0]
li_list = general_section.find_all("li")

# 成对提取字段名和值：偶数索引是字段，奇数是对应值
data_dict = {}
for i in range(0, len(li_list), 2):
    field = li_list[i].get_text(strip=True)
    value = li_list[i+1].get_text(strip=True)
    if field in cols:
        data_dict[field] = value

# 处理Activity Information板块
activity_section = htmlSoup.select("div.bs-docs-section")[1]
activity_li = activity_section.find_all("li")
for i in range(0, len(activity_li), 2):
    field = activity_li[i].get_text(strip=True)
    value = activity_li[i+1].get_text(strip=True)
    if field in cols:
        data_dict[field] = value

# 转换为DataFrame行
new_table = pd.concat([new_table, pd.DataFrame([data_dict])], ignore_index=True)
print(new_table)

二、实现批量爬取 #

2.1 单列表页提取20条详情链接 #

列表页URL通过page参数控制页码，提取当前页所有详情链接的函数：

def get_detail_links(page_num):
    # 复用你提供的搜索链接，替换页码参数
    list_url = f'http://dramp.cpu-bioinfor.org/search/advanced_search.php?page={page_num}&geneinfo_data%5B0%5D=&boo_gene%5B0%5D=And&geneinfo_data%5B1%5D=&boo_gene%5B1%5D=And&length=&boo_length=And&geneinfo_data%5B2%5D=&boo_gene%5B2%5D=And&geneinfo_data%5B3%5D=&boo_gene%5B3%5D=And&geneinfo_data%5B4%5D=&boo_gene%5B4%5D=And&ckbx1%5B%5D=&ckbx1%5B%5D=Antimicrobial&boo_act=And&activity%5B0%5D=&bool_cactivity%5B0%5D=And&comments%5B0%5D=&bool_comments%5B0%5D=And&comments%5B1%5D=&bool_comments%5B1%5D=And&db=&db_id='
    response = rq.get(list_url)
    soup = BeautifulSoup(response.content, "lxml")
    detail_links = []
    # 跳过表头行，遍历数据行提取详情链接
    rows = soup.select("table.table tr")[1:]
    for row in rows:
        a_tag = row.find("a")
        if a_tag:
            detail_id = a_tag.get('href').split('id=')[-1]
            full_link = f'{base_url}{detail_id}'
            detail_links.append(full_link)
    return detail_links

2.2 逐页爬取到第285页 #

结合详情页提取逻辑，批量爬取所有页面：

import time

# 初始化总数据表格
total_df = pd.DataFrame(columns=cols)

# 遍历1到285页
for page in range(1, 286):
    print(f'正在爬取第{page}页...')
    try:
        # 获取当前页所有详情链接
        detail_links = get_detail_links(page)
        # 逐个爬取详情页数据
        for link in detail_links:
            time.sleep(1)  # 添加延迟，避免触发反爬机制
            response = rq.get(link)
            soup = BeautifulSoup(response.content, "lxml")
            data_dict = {}
            
            # 提取General板块数据
            general_section = soup.select("div.bs-docs-section")[0]
            li_list = general_section.find_all("li")
            for i in range(0, len(li_list), 2):
                field = li_list[i].get_text(strip=True)
                value = li_list[i+1].get_text(strip=True)
                if field in cols:
                    data_dict[field] = value
            
            # 提取Activity板块数据
            activity_section = soup.select("div.bs-docs-section")[1]
            activity_li = activity_section.find_all("li")
            for i in range(0, len(activity_li), 2):
                field = activity_li[i].get_text(strip=True)
                value = activity_li[i+1].get_text(strip=True)
                if field in cols:
                    data_dict[field] = value
            
            # 添加到总表格
            total_df = pd.concat([total_df, pd.DataFrame([data_dict])], ignore_index=True)
    except Exception as e:
        print(f'第{page}页爬取失败：{str(e)}')
        continue

# 保存结果到CSV文件
total_df.to_csv('dramp_data.csv', index=False, encoding='utf-8-sig')
print('爬取完成，数据已保存到dramp_data.csv')

内容的提问来源于stack exchange，提问作者S.EB