修复BeautifulSoup爬虫代码 实现多页数据采集并导出CSV

递归调用分页采集逻辑时，未将后续页面返回的采集数据合并到当前页的数据集内，导致函数最终仅返回第一页的采集结果。

import requests
from bs4 import BeautifulSoup
import pandas as pd

base_url = 'https://www.marketresearch.com'
start_url = 'https://www.marketresearch.com/search/results.asp?qtype=2&datepub=3&publisher=Technavio&categoryid=0&sortby=r'

# 配置请求头模拟浏览器，避免被反爬拦截
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
}

def scrape_it(url):
    page = requests.get(url, headers=headers)
    soup = BeautifulSoup(page.text, 'html.parser')
    next_ele = soup.find_all(class_="standardLinkDkBlue")[-1]
    # 拼接完整的下一页链接，处理相对路径问题
    nexturl = base_url + next_ele['href']
    stri = next_ele.string
    reports = soup.find_all("tr", {"class": ["SearchTableRowAlt", "SearchTableRow"]})
    data = []

    for report in reports:
        data.append({
               'title': report.find('a', class_='linkTitle').text,
               'price': report.find('div', class_='resultPrice').text,
               'date_author': report.find('div', class_='textGrey').text.replace(' | published by: TechNavio', ''),
               'detail_link': report.a['href']
        })

    if 'next' not in stri:
        print("All pages completed")
    else:
        # 核心修改：将后续页面的采集结果合并到当前数据集
        data.extend(scrape_it(nexturl))

    return data

myOutput = pd.DataFrame(scrape_it(start_url))
myOutput.to_csv('results-tec6.csv', header=False, encoding='utf-8-sig')

• 核心修复：原代码递归调用scrape_it(nexturl)时未接收返回值，修改为data.extend(scrape_it(nexturl))，将所有后续页面的采集结果追加到当前数据集，最终返回全量数据
• 新增请求头配置：模拟浏览器请求，降低被网站反爬策略拦截的概率
• 补充链接拼接逻辑：处理分页链接为相对路径的问题，保证下一页请求正常发送
• 导出CSV新增encoding='utf-8-sig'参数，避免中文乱码问题

内容的提问来源于stack exchange，提问作者Michael Wiz