如何将爬取的多页分页表格合并为单个Pandas DataFrame？

你的问题出在每次循环都用新页面的数据覆盖了df变量，导致最终只保留最后一页的结果。要合并所有页面数据，只需要把每一页的DataFrame暂存到列表中，最后统一合并即可。

修改后的代码如下：

import requests
import pandas as pd
from bs4 import BeautifulSoup


data = {
    "action": "geteCMSList",
    "keyword": "",
    "officeId": "0",
    "contractAwardTo": "",
    "contractStartDtFrom": "",
    "contractStartDtTo": "",
    "contractEndDtFrom": "",
    "contractEndDtTo": "",
    "departmentId": "",
    "tenderId": "",
    "procurementMethod": "",
    "procurementNature": "",
    "contAwrdSearchOpt": "Contains",
    "exCertSearchOpt": "Contains",
    "exCertificateNo": "",
    "tendererId": "",
    "procType": "",
    "statusTab": "eTenders",
    "pageNo": "1",
    "size": "10",
    "workStatus": "All",
}


_columns = [
    "S. No",
    "Ministry, Division, Organization, PE",
    "Procurement Nature, Type & Method",
    "Tender/Proposal ID, Ref No., Title..",
    "Contract Awarded To",
    "Company Unique ID",
    "Experience Certificate No  ",
    "Contract Amount",
    "Contract Start & End Date",
    "Work Status",
]

# 初始化空列表存储每一页的DataFrame
dfs = []

for page in range(1, 11):  # 调整这里的页数范围
    print(f"Page: {page}")
    data["pageNo"] = page

    response = requests.post(
        "https://www.eprocure.gov.bd/AdvSearcheCMSServlet", data=data
    )
    # 修复缺失的table标签
    soup = BeautifulSoup("<table>" + "".join(response.text) + "</table>", "html.parser")
    page_df = pd.read_html(str(soup))[0]

    page_df.columns = _columns
    print(page_df.to_string())
    # 将当前页的DataFrame添加到列表
    dfs.append(page_df)

# 合并所有页面的数据到单个DataFrame
final_df = pd.concat(dfs, ignore_index=True)
# 可以查看合并后的结果
print("\n合并后的数据总行数:", len(final_df))
print(final_df.to_string())

关键改动说明 #

• 新增dfs = []：创建一个空列表，用来收集每一页爬取到的DataFrame
• 把原代码中的df改为page_df：避免每次循环覆盖变量，而是将单页数据存入临时变量
• dfs.append(page_df)：将当前页的DataFrame添加到列表中
• 循环结束后用pd.concat(dfs, ignore_index=True)：把列表中所有DataFrame纵向合并，ignore_index=True会重置合并后的索引，避免出现重复的行号

内容的提问来源于stack exchange，提问作者Nazmul