使用Beautiful Soup的Next Sibling提取拍卖剩余时间遇阻求助

问题原因 #

你之前的三种方法全部失效，核心原因是这个倒计时是JavaScript动态渲染生成的。服务器返回的原始HTML里，tzcd这类节点并没有实际的倒计时文本内容，只有浏览器加载并执行页面JS后，才会把实时剩余时间填充到节点里。而BeautifulSoup只能解析静态HTML，无法处理JS动态生成的内容，所以拿不到有效数据。

解决方案 #

方案1：使用支持JS渲染的爬虫工具（新手推荐） #

用selenium或playwright模拟浏览器加载页面，等JS渲染完成后再提取元素。

示例代码（Selenium）

先安装依赖：

pip install selenium

同时需要下载对应浏览器的驱动（比如ChromeDriver，版本要和你的Chrome浏览器匹配），放到系统PATH里或指定路径。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv

# 初始化Chrome浏览器
driver = webdriver.Chrome()
target_url = "https://auctionofchampions.com/Michael_Jordan___Magic_Johnson_Signed_Lmt__Ed__Pho-LOT271177.aspx"
driver.get(target_url)

try:
    # 等待倒计时元素加载完成（最多等10秒）
    time_element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "tzcd"))
    )
    # 提取并清理时间文本
    time_left = time_element.text.replace("Time Left:", "").strip()
    print(f"提取到的剩余时间：{time_left}")

    # 写入CSV文件
    with open("auction_info.csv", "w", newline="", encoding="utf-8") as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow(["剩余时间"])
        writer.writerow([time_left])
finally:
    # 关闭浏览器
    driver.quit()

示例代码（Playwright）

Playwright无需手动下载驱动，配置更简单：

pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright
import csv

with sync_playwright() as p:
    # 启动浏览器（headless=False可看到窗口，方便调试）
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    page.goto("https://auctionofchampions.com/Michael_Jordan___Magic_Johnson_Signed_Lmt__Ed__Pho-LOT271177.aspx")
    
    # 提取倒计时文本
    time_left = page.locator("#tzcd").text_content().replace("Time Left:", "").strip()
    print(f"提取到的剩余时间：{time_left}")

    # 写入CSV
    with open("auction_info.csv", "w", newline="", encoding="utf-8") as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow(["剩余时间"])
        writer.writerow([time_left])
    
    browser.close()

方案2：抓包获取API接口（进阶高效） #

打开浏览器开发者工具（F12），切换到「Network」标签页，刷新页面，搜索time「countdown」等关键词，找到返回倒计时数据的XHR/JSON接口。直接用requests请求该接口解析数据，无需模拟浏览器，效率更高。

示例代码（需替换实际接口信息）：

import requests
import csv

# 实际接口URL和字段需抓包确认
api_url = "https://auctionofchampions.com/api/get-countdown?lot_id=271177"
response = requests.get(api_url)
data = response.json()

time_left = data["timeLeft"]  # 替换为接口返回的实际字段名
print(f"剩余时间：{time_left}")

# 写入CSV
with open("auction_info.csv", "w", newline="", encoding="utf-8") as csv_file:
    writer = csv.writer(csv_file)
    writer.writerow(["剩余时间"])
    writer.writerow([time_left])

内容的提问来源于stack exchange，提问作者MyVeblen