使用Selenium爬取Circle Chart网站时销量无法写入CSV问题求助

尝试爬取CircleChart 2012年年度专辑榜的歌手名、专辑名、专辑销量，但使用以下Selenium代码时，仅能获取歌手名和专辑名，销量数据无法写入CSV文件，预期CSV需包含这三个字段。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv

url = 'https://circlechart.kr/page_chart/album.circle?nationGbn=T&targetTime=2012&hitYear=2012&termGbn=year&yearTime=3'

# Initialize WebDriver
driver = webdriver.Chrome()
driver.get(url)

# Wait until the table is loaded
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'ChartTablePC')))

# Locate the table
table = driver.find_element(By.CLASS_NAME, 'ChartTablePC')
rows = table.find_elements(By.TAG_NAME, 'tr')

# Print structure of each row to verify
for row in rows[:5]:  # Print the first 5 rows for inspection
    cells = row.find_elements(By.TAG_NAME, 'td')
    print([cell.text for cell in cells])

# Define headers
HEADERS = ['Rank', 'Album Artist', 'Sales']

# Open CSV file for writing
with open('EXOalbumSales2012.csv', 'w', newline='', encoding='utf-8') as outfile:
    writer = csv.writer(outfile)
    writer.writerow(HEADERS)

    # Iterate over rows
    for row in rows[1:]:  # Skip the header row
        cells = row.find_elements(By.TAG_NAME, 'td')
        if len(cells) >= 5:  # Ensure there are enough columns
            rank = cells[0].text.strip()
            album_artist = cells[2].text.strip()
            sales = cells[4].text.strip()
            
            data_out = [rank, album_artist, sales]
            writer.writerow(data_out)
            print(data_out)

# Close the WebDriver
driver.quit()

问题原因及修正方案 #

1. 元素加载时机问题：仅用presence_of_element_located只能确保元素存在，但文本可能还未渲染完成，导致获取到空值。需改用visibility_of_element_located等待表格完全可见。
2. 单元格索引错误：原代码中销量取cells[4]，但实际页面中销量字段的索引为cells[5]，需调整索引位置。
3. 缺少滚动可见处理：部分行可能在页面视窗外，未触发文本渲染，需滚动到行元素位置确保内容加载。
4. 字段匹配问题：原headers未包含专辑名，需调整字段顺序以匹配需求。

修正后的代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import csv

url = 'https://circlechart.kr/page_chart/album.circle?nationGbn=T&targetTime=2012&hitYear=2012&termGbn=year&yearTime=3'

driver = webdriver.Chrome()
driver.get(url)

# 等待表格完全可见，确保文本已渲染
WebDriverWait(driver, 15).until(EC.visibility_of_element_located((By.CLASS_NAME, 'ChartTablePC')))

table = driver.find_element(By.CLASS_NAME, 'ChartTablePC')
rows = table.find_elements(By.TAG_NAME, 'tr')

# 确认单元格结构（保留用于调试）
for row in rows[:5]:
    cells = row.find_elements(By.TAG_NAME, 'td')
    print([cell.text for cell in cells])

# 调整headers为需求的字段
HEADERS = ['排名', '专辑名', '歌手', '销量']

with open('EXOalbumSales2012.csv', 'w', newline='', encoding='utf-8') as outfile:
    writer = csv.writer(outfile)
    writer.writerow(HEADERS)

    for row in rows[1:]:
        cells = row.find_elements(By.TAG_NAME, 'td')
        # 确保单元格数量足够（实际页面每行有7个td）
        if len(cells) >= 6:
            # 滚动到当前行，确保内容加载
            driver.execute_script("arguments[0].scrollIntoView();", row)
            
            rank = cells[0].text.strip()
            album_name = cells[1].text.strip()  # 专辑名在第2个单元格
            artist = cells[2].text.strip()     # 歌手在第3个单元格
            sales = cells[5].text.strip()      # 销量在第6个单元格
            
            data_out = [rank, album_name, artist, sales]
            writer.writerow(data_out)
            print(data_out)

driver.quit()

内容的提问来源于stack exchange，提问作者Zoeyyyy