BeautifulSoup：Python循环仅执行8次后以代码0退出（VS Code）

核心问题原因 #

• 静态HTML仅返回前8条数据：Trademe搜索结果采用动态加载，requests.get()只能获取页面初始渲染的8条内容，后续列表需要JavaScript异步加载，导致解析到的结果只有前8条。
• ID判断逻辑错误：将数据库查询结果转为字符串后判断auctionID not in allResult，会出现误判（比如ID12会被误认为包含在123的字符串中），导致后续合法ID被错误判定为已存在。
• 冗余循环风险：内层zip(title, location, image)循环完全多余（每个列表项的对应元素都是唯一的），且若某元素获取失败，zip会直接终止循环，导致该条数据无法入库。

修复步骤 #

1. 修正数据库ID判断逻辑 #

将查询到的ID提取为集合，利用集合的高效查询避免误判：

c.execute('''SELECT ID FROM trademe ORDER BY DateAdded DESC ''')
existing_ids = {row[0] for row in c.fetchall()}

后续判断改为：

if auctionID not in existing_ids:

2. 获取完整页面数据 #

由于requests无法执行JS，可通过两种方式获取完整结果：

• 调用API：通过浏览器开发者工具的网络面板，找到Trademe的搜索接口（一般是XHR类型的请求），直接用requests调用API获取JSON格式的完整数据。
• 用Selenium模拟浏览器：等待页面加载完成后再解析，示例代码如下：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get(url)
try:
    # 等待10秒，直到所有列表项加载完成
    WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".tm-marketplace-search-card__wrapper"))
    )
except:
    pass
soup = BeautifulSoup(driver.page_source, "html.parser")
driver.quit()

3. 简化循环逻辑并优化资源管理 #

• 去掉冗余的内层循环，直接提取单个元素的内容
• 操作完成后关闭数据库连接，避免资源泄漏
• 新增ID后更新已存在集合，避免重复查询数据库

完整修复代码 #

from bs4 import BeautifulSoup
from time import sleep
import requests
import datetime
import sqlite3

def mechanicalKeyboards():
    url = "https://www.trademe.co.nz/a/marketplace/computers/peripherals/keyboards/mechanical/search?condition=used&sort_order=expirydesc"
    dateAdded = datetime.datetime.now().strftime("%d/%m/%Y %H:%M:%S")
    
    # 用Selenium获取完整页面
    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By
    
    driver = webdriver.Chrome()
    driver.get(url)
    try:
        WebDriverWait(driver, 10).until(
            EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".tm-marketplace-search-card__wrapper"))
        )
    except:
        pass
    soup = BeautifulSoup(driver.page_source, "html.parser")
    driver.quit()
    
    listingContainer = soup.select(".tm-marketplace-search-card__wrapper")
    conn = sqlite3.connect('trademe.db')
    c = conn.cursor() 
    # 修正ID判断逻辑
    c.execute('''SELECT ID FROM trademe ORDER BY DateAdded DESC ''')
    existing_ids = {row[0] for row in c.fetchall()}
    
    for listing in listingContainer:
        # 提取单个元素，处理可能的空值
        title_elem = listing.select_one("#-title")
        title = title_elem.text.strip() if title_elem else "No Title"
        
        auctionID = listing['data-aria-id'].split("-").pop()
        fullListingURL = f"https://www.trademe.co.nz/a/{auctionID}"
        
        # 处理价格
        buyNow = "None"
        price = "None"
        pricing_row = listing.select_one(".tm-marketplace-search-card__footer-pricing-row")
        if pricing_row:
            buy_now_elem = pricing_row.find(class_="tm-marketplace-search-card__price ng-star-inserted")
            buyNow = buy_now_elem.text.strip() if buy_now_elem else "None"
            price_elem = pricing_row.find(class_="tm-marketplace-search-card__price")
            price = price_elem.text.strip() if price_elem else "None"
        
        # 准确判断是否入库
        if auctionID not in existing_ids:
            print(f"Adding new data - {title}")
            c.execute(''' INSERT INTO trademe VALUES(?,?,?,?)''', (auctionID, title, dateAdded, fullListingURL))
            conn.commit()
            existing_ids.add(auctionID)
            sleep(5)
    
    conn.close()

mechanicalKeyboards()

额外建议 #

• 添加User-Agent等请求头，模拟真实浏览器请求，降低反爬风险
• 用上下文管理器（with conn:）处理数据库连接，自动管理提交和关闭
• 增加全局异常捕获，避免单个列表项解析失败导致脚本终止

内容的提问来源于stack exchange，提问作者Spykerwolf