Python多页爬取问题：循环仅获取第一页数据如何解决？

问题描述 #

我是Python新手，正在完成大学课程作业，尝试用Python Pandas爬取网站https://aaiasb.gr/publications/investigation-reports的多页表格数据。已获取所有页面URL，但执行循环爬取代码时仅得到第一页数据，请求帮助解决。

原代码如下：

#imports
from bs4 import BeautifulSoup
import requests
import time

import pandas as pd


pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
response=requests.get('https://aaiasb.gr/publications/investigation-reports', ) 
response

soup = BeautifulSoup(response.text, 'html.parser')
soup 

base_url = 'https://aaiasb.gr/publications/investigation-reports'
ending = '?start='
numbers = [50, 100, 150]

urls = [base_url]
for n in numbers:
    url = base_url+ending+str(n)
    urls.append(url)

df = pd.DataFrame(urls) 
df = df.rename(columns={df.columns[0]:'url'})
df

for url in urls:


    response = requests.get(url)
    time.sleep(3)
    soup_doc = BeautifulSoup(response.text, 'html.parser')



    entries = []

    page=soup.select('div.cck_page_items')[0]

    rows = page.find('table').find_all('tr')[1:] 
    

    conclusion_date1 = tr.find_all('td')[0].find_all('div')[1].text.strip()


    conclusion_date2 = tr.find_all('td')[0].find_all('div')[2].text.strip()


    incident_info = tr.find_all('td')[1].find_all('div')[0].text.strip()

    incident_type = tr.find_all('td')[1].find_all('div')[1].text.strip()

    incident_description = str(tr.find_all('td')[1].find_all('span', attrs={'uk-icon':'info'})[0])
    fatalities = tr.find_all('td')[1].find_all('div')[2].text.strip()

    fatalities_description = str(tr.find_all('td')[1].find_all('span', attrs={'uk-icon':'info'})[1])

    area = tr.find_all('td')[2].find_all('div')[0].text.strip()

    registry = tr.find_all('td')[2].find_all('div')[1].text.strip()
    
    aircraft_type = tr.find_all('td')[2].find_all('div')[-2].text.strip()

    aircraft_info = tr.find_all('td')[2].find_all('div')[-1].text.strip()


    area_info = tr.find_all('td')[2].text.strip()


    dict = {'conclusion_date1': conclusion_date1,
        'conclusion_date2': conclusion_date2,
        'incident_info': incident_info,
        'incident_type': incident_type,
        'incident_description': incident_description,
        'fatalities': fatalities,
        'fatalities_description': fatalities_description,
        'area': area,
        'registry': registry,
        'aircraft_type': aircraft_type,
        'aircraft_info': aircraft_info,
        'area_info': area_info}

    entries.append(dict)
     
df1 =pd.DataFrame(entries)

错误原因分析 #

1. 重复使用初始页面解析对象：循环内始终调用一开始的soup（第一页的解析结果），而非当前页面的soup_doc，导致每次都爬取第一页内容
2. 数据列表被重复初始化：entries列表在循环内部定义，每次循环都会清空之前收集的数据
3. 缺少行遍历逻辑：代码直接使用tr变量但未定义，实际运行会报错，且没有遍历表格的每一行数据

修正后的代码 #

# imports
from bs4 import BeautifulSoup
import requests
import time
import pandas as pd

pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)

base_url = 'https://aaiasb.gr/publications/investigation-reports'
ending = '?start='
numbers = [50, 100, 150]

# 生成所有页面URL
urls = [base_url]
for n in numbers:
    url = f"{base_url}{ending}{n}"
    urls.append(url)

# 初始化存储所有数据的列表（放在循环外，避免重复清空）
entries = []

for url in urls:
    response = requests.get(url)
    time.sleep(3)
    # 使用当前页面的解析对象soup_doc
    soup_doc = BeautifulSoup(response.text, 'html.parser')
    
    # 获取当前页面的表格区域
    page = soup_doc.select('div.cck_page_items')[0]
    rows = page.find('table').find_all('tr')[1:]  # 跳过表头行
    
    # 遍历当前页面的每一行数据
    for tr in rows:
        try:
            conclusion_date1 = tr.find_all('td')[0].find_all('div')[1].text.strip()
            conclusion_date2 = tr.find_all('td')[0].find_all('div')[2].text.strip()
            
            incident_info = tr.find_all('td')[1].find_all('div')[0].text.strip()
            incident_type = tr.find_all('td')[1].find_all('div')[1].text.strip()
            # 处理可能不存在的span元素，避免索引错误
            incident_desc_span = tr.find_all('td')[1].find_all('span', attrs={'uk-icon':'info'})
            incident_description = str(incident_desc_span[0]) if len(incident_desc_span) > 0 else ""
            
            fatalities = tr.find_all('td')[1].find_all('div')[2].text.strip()
            fatalities_desc_span = tr.find_all('td')[1].find_all('span', attrs={'uk-icon':'info'})
            fatalities_description = str(fatalities_desc_span[1]) if len(fatalities_desc_span) > 1 else ""
            
            area = tr.find_all('td')[2].find_all('div')[0].text.strip()
            registry = tr.find_all('td')[2].find_all('div')[1].text.strip()
            aircraft_type = tr.find_all('td')[2].find_all('div')[-2].text.strip()
            aircraft_info = tr.find_all('td')[2].find_all('div')[-1].text.strip()
            area_info = tr.find_all('td')[2].text.strip()
            
            # 将当前行数据存入字典
            row_dict = {
                'conclusion_date1': conclusion_date1,
                'conclusion_date2': conclusion_date2,
                'incident_info': incident_info,
                'incident_type': incident_type,
                'incident_description': incident_description,
                'fatalities': fatalities,
                'fatalities_description': fatalities_description,
                'area': area,
                'registry': registry,
                'aircraft_type': aircraft_type,
                'aircraft_info': aircraft_info,
                'area_info': area_info
            }
            entries.append(row_dict)
        except IndexError:
            # 处理可能的结构异常行，避免程序崩溃
            print(f"处理行时出错，URL: {url}")
            continue

# 将所有累积的数据转为DataFrame
df1 = pd.DataFrame(entries)
print(df1)

关键修改点 #

• 将entries列表初始化移至循环外，确保所有页面的数据能累积存储
• 替换page=soup.select(...)为page=soup_doc.select(...)，使用当前页面的解析结果
• 添加for tr in rows:循环，遍历表格的每一行数据
• 增加try-except异常捕获，避免因页面结构不一致导致程序崩溃
• 优化元素获取逻辑，处理可能不存在的元素，避免索引错误

内容的提问来源于stack exchange，提问作者Airin