爬虫导出Excel时两款机车产品（63256/69256）数据缺失排查求助

我开发了一段代码用于爬取Roco网站上的蒸汽机车数据，但导出到Excel时，产品63256和69256未在文件中显示（对应Excel第6、7行），求排查思路。

相关代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd
import xlsxwriter

baseurl = 'https://www.roco.cc/'

headers = {
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
}

productlinks = []

for x in range(1,2):
    r = requests.get(
        f'https://www.roco.cc/ren/products/locomotives/steam-locomotives.html?p={x}&verfuegbarkeit_status=41%2C42%2C43%2C45%2C44')
    soup = BeautifulSoup(r.content, 'lxml')
    productlist = soup.find_all('li', class_='item product product-item')
    
    for item in productlist:
        for link in item.find_all('a', class_='product-item-link', href=True):
            productlinks.append(link['href'])

Loco_list = []
Spare_parts_list = []

for link in productlinks:
    r = requests.get(link, allow_redirects=False)
    soup = BeautifulSoup(r.content, 'lxml')
    try:
        Manufacturer_name = soup.find(
            'div', class_='product-head-name').h1.text.strip()
    except:
        Manufacturer_name = ''
    try:
        Reference = soup.find('span', class_='product-head-artNr').text.strip()
    except:
        Reference = ''
        
    try:  
        Price = soup.find('div', class_='product-head-price').text.strip()
    except:
        Price = ''
        
    Type = 'Steam locomotive'
        
    try:
        Scale = soup.find('td', {'data-th': 'Scale'}).text.strip()
    except:
        Scale = ''

    try:  
        Current = soup.find('td', {'data-th': 'Control'}).text.split(' ')[0]
    except:
        Current = ''


    try:  
        Control = soup.find('td', {'data-th': 'Control'}).text.strip()
    except:
        Control = ''

    try:
        Interface = soup.find('td', {'data-th': 'Interface'}).text.strip()
    except:
        Interface = ''

    try:
        Digital_decoder = soup.find(
            'td', {'data-th': 'Digital decoder'}).text.strip()
    except:
        Digital_decoder = ''

    try:
        Decoder_Type = soup.find(
            'td', {'data-th': 'Decoder-Type'}).text.strip()
    except:
        Decoder_Type = ''

    try:
        Motor = soup.find('td', {'data-th': 'Motor'}).text.strip()
    except:
        Motor = ''

    try:
        Flywheel = soup.find('td', {'data-th': 'Flywheel'}).text.strip()
    except:
        Flywheel = ''

    try:
        Minimum_radius = soup.find(
            'td', {'data-th': 'Minimum radius'}).text.strip()
    except:
        Minimum_radius = ''

    try:
        Length_over_buffer = soup.find(
            'td', {'data-th': 'Length over buffer'}).text.strip()
    except:
        Length_over_buffer = ''

    try:
        Number_of_driven_axles = soup.find(
        'td', {'data-th': 'Number of  driven axles'}).text.strip()
    except:
        Number_of_driven_axles = ''

    try:
        Number_of_axles_with_traction_tyres = soup.find(
        'td', {'data-th': 'Number of  axles with traction tyres'}).text.strip()
    except:
        Number_of_axles_with_traction_tyres = ''

    try:
        Coupling = soup.find('td', {'data-th': 'Coupling'}).text.strip()
    except:
        Coupling = ''

    try:
        LED_lighting = soup.find(
            'td', {'data-th': 'LED lighting'}).text.strip()
    except:
        LED_lighting = ''

    try:
        Head_light = soup.find('td', {'data-th': 'Head light'}).text.strip()
    except:
        Head_light = ''

    try:
        LED_head_light = soup.find(
            'td', {'data-th': 'LED head light'}).text.strip()
    except:
        LED_head_light = ''

    try:
        Country = soup.find(
            'td', {'data-th': 'Original (country)'}).text.strip()
    except:
        Country = ''

    try:
        Railway_company = soup.find(
            'td', {'data-th': 'Railway Company'}).text.strip()
    except:
        Railway_company = ''

    try:
        Epoch = soup.find('td', {'data-th': 'Epoch'}).text.strip()
    except:
        Epoch = ''

    try:
        Description = soup.find(
            'div', class_='product-add-form-text').text.strip()
    except:
        Description = ''


    Locomotives = {
        'Manufacturer_name': Manufacturer_name,
        'Reference': Reference,
        'Price': Price,
        'Type': Type,
        'Scale': Scale,
        'Current': Current,
        'Control': Control,
        'Interface': Interface,
        'Digital_decoder': Digital_decoder,
        'Decoder_Type': Decoder_Type,
        'Motor': Motor,
        'Flywheel': Flywheel,
        'Minimum_radius': Minimum_radius,
        'Length_over_buffer': Length_over_buffer,
        'Number_of_driven_axles': Number_of_driven_axles,
        'Number_of_axles_with_traction_tyres': Number_of_axles_with_traction_tyres,
        'Coupling': Coupling,
        'LED_lighting': LED_lighting,
        'Head_light': Head_light,
        'LED_head_light': LED_head_light,
        'Country': Country,
        'Railway_company': Railway_company,
        'Epoch': Epoch,
        'Description': Description,
    } 

    Loco_list.append(Locomotives)

print(Locomotives)

df1 = pd.DataFrame(Loco_list)
writer = pd.ExcelWriter('Roco - locomotives.xlsx', engine='xlsxwriter')
df1.to_excel(writer, sheet_name='Model')
writer.save()

print('Saved to file')

排查思路 #

• 确认产品链接是否被抓取
  打印productlinks列表，检查63256、69256对应的URL是否存在。如果不存在：

  • 检查这两款产品在列表页的HTML结构，是否和其他产品使用相同的item product product-item类，或是product-item-link的class有差异
  • 核对列表页请求参数verfuegbarkeit_status，确认这两款产品的库存状态是否在指定的41/42/43/45/44范围内，是否被过滤

• 检查产品页请求状态
  如果链接已在productlinks中，单独请求这两个产品的URL，查看响应状态码。代码中设置了allow_redirects=False，若页面存在跳转，会导致无法获取内容，进而所有字段为空（但仍会加入列表，需检查Excel是否有空行）

• 对比产品页HTML结构差异
  手动打开两款缺失产品的页面，和正常产品页对比：

  • 确认product-head-artNr等关键元素的class是否存在，是否有替代class
  • 检查表格中data-th属性值是否一致（比如是否存在空格、大小写差异）
  • 验证product-add-form-text容器是否存在，描述内容是否存放在其他位置

• 检查异常捕获后的逻辑
  在循环中打印每个产品的Reference值，确认这两款产品的编号是否能正常获取。若为空，说明对应字段的解析逻辑不匹配页面结构

• 验证DataFrame导出环节
  导出前打印df1的长度和内容，确认两款产品的数据是否在DataFrame中。若存在但Excel中未显示，需检查是否有特殊字符导致导出失败，或是行号对应问题（注意DataFrame默认带索引，Excel第6行对应索引5）

内容的提问来源于stack exchange，提问作者Filip Chobodicky