Python网页爬虫脚本遇IndexError问题求助（附代码及报错）

错误原因 #

你遍历的所有govuk-grid-row类的div中，并非每个都包含govuk-body类的子元素，直接通过[0]索引访问空列表就会触发IndexError。同时当前代码还存在隐性问题：如果某个条目只有名称没有地址，会导致name、address1、address2三个列表长度不一致，后续生成DataFrame时必然报错。

修复后的代码 #

from bs4 import BeautifulSoup
import requests
import pandas as pd
from time import sleep
from random import randint

# 初始化存储列表
name = []
address1 = []
address2 = []


def getPageResults(postcode, page):
    url = 'https://www.ukrlp.co.uk/ukrlp/ukrlp_provider.page_pls_searchProviders'
    params = {
        'pv_pc': postcode,
        'pn_pc_d': 5,  # 距离：0,1,5,10英里
        'pn_pageNo': str(page),
        'pv_layout': 'SEARCH'
    }
    
    # 用requests替代urlopen，参数传递更清晰
    response = requests.get(url, params=params)
    response.encoding = 'utf-8'
    soup = BeautifulSoup(response.text, "html.parser")

    results_found = False

    for row in soup.find_all("div", class_="govuk-grid-row"):
        # 先获取名称元素，只处理有名称的有效结果行
        name_elem = row.select_one("h2")
        address_elems = row.select("div.govuk-body")
        
        if name_elem:
            results_found = True
            name.append(name_elem.get_text(strip=True))
            
            # 处理地址，无对应元素时填充空字符串
            addr1 = address_elems[0].get_text(strip=True) if len(address_elems) >= 1 else ""
            addr2 = address_elems[1].get_text(strip=True) if len(address_elems) >= 2 else ""
            address1.append(addr1)
            address2.append(addr2)

    return results_found


print("__________________________")

postcodes = [
    "L31"
    # 其他邮编可在此处添加
]

for postcode in postcodes:
    print(f"处理邮编: {postcode}")
    page = 1
    while True:
        has_results = getPageResults(postcode, page)
        sleep(randint(1, 3))
        
        if not has_results:
            break
        page += 1
        print(f"正在处理第 {page} 页")

# 生成DataFrame并保存
df = pd.DataFrame({
    "name": name,
    "address1": address1,
    "address2": address2
})

df.to_excel("Five_miles_9.xlsx", index=False)

关键修复点 #

• 避免索引越界：先检查地址元素列表的长度，再访问对应索引，无地址时填充空字符串
• 过滤无效行：只处理包含名称的结果行，跳过页面中其他govuk-grid-row格式的非内容行
• 保证数据一致性：每个有效条目同时向三个列表添加内容，避免DataFrame生成时因长度不匹配报错
• 代码优化：用requests简化请求逻辑，用select_one获取单个元素提升效率，变量命名更具可读性

内容的提问来源于stack exchange，提问作者Ren