如何用Selenium Python的find_element删除爬取表格中的空白值

问题说明 #

爬取CME GROUP目标页面时，获取到的元素列表中MONTH列（对应代码里的Date字段）有一半是空白值，但Price、Updated等其他列均能正常获取无空值。目标页面为：https://www.cmegroup.com/markets/energy/refined-products/gasoil-01-rotterdam-barges-swap.quotes.html

原爬取代码 #

main = driver.find_element(By.ID, "main-content")
time.sleep(10)
matches = main.find_elements(By.XPATH,
                          '//*[@id="productTabData"]/div/div/div/div/div/div[2]/div/div/div/div/div/div[5]/div/div/div/div[1]/div/table/tbody/tr')
time.sleep(10)
dane = []
for match in matches:
    Date = match.find_element(By.XPATH, "./td[1]/div/span/b").text
    Price = match.find_element(By.XPATH, "./td[5]").text
    Updated = match.find_element(By.XPATH, "./td[10]").text
    print(Date)
    table = {
        "DataPL" : Date,
        "GO" : Price,
        "Updated" : Updated
    }

    dane.append(table)
df=pd.DataFrame(dane)

临时解决方案（Pandas Shift） #

目前通过Pandas的.shift()方法临时处理，但希望找到更优方案，临时代码如下：

df["DataPL"] = df["DataPL"].shift(-18)
df = df.iloc[0:17,:2]

更优解决方案 #

问题根源是目标表格的MONTH列采用了合并单元格布局：只有每组数据的第一行包含实际月份文本，后续行的该单元格为空。以下是两种更合理的处理方式：

方式1：复用最近有效月份值 #

遍历表格行时，若当前行的MONTH元素无文本，则复用最近一次获取到的有效月份值：

main = driver.find_element(By.ID, "main-content")
time.sleep(10)
matches = main.find_elements(By.XPATH,
                          '//*[@id="productTabData"]/div/div/div/div/div/div[2]/div/div/div/div/div/div[5]/div/div/div/div[1]/div/table/tbody/tr')
time.sleep(10)
dane = []
current_month = ""  # 存储最近一次有效的月份值

for match in matches:
    date_elem = match.find_element(By.XPATH, "./td[1]/div/span/b")
    date_text = date_elem.text.strip()
    
    # 更新或复用月份值
    if date_text:
        current_month = date_text
    Date = current_month
    
    Price = match.find_element(By.XPATH, "./td[5]").text.strip()
    Updated = match.find_element(By.XPATH, "./td[10]").text.strip()
    
    table = {
        "DataPL": Date,
        "GO": Price,
        "Updated": Updated
    }
    dane.append(table)

df = pd.DataFrame(dane)

方式2：按分组定位数据行 #

直接定位所有包含MONTH文本的行，再获取该组下的所有关联数据行：

main = driver.find_element(By.ID, "main-content")
time.sleep(10)
# 定位所有带有月份文本的行（合并单元格的首行）
month_rows = main.find_elements(By.XPATH,
                          '//*[@id="productTabData"]/div/div/div/div/div/div[2]/div/div/div/div/div/div[5]/div/div/div/div[1]/div/table/tbody/tr[./td[1]/div/span/b]')
time.sleep(10)
dane = []

for month_row in month_rows:
    current_month = month_row.find_element(By.XPATH, "./td[1]/div/span/b").text.strip()
    
    # 处理当前月份行本身的数据
    price = month_row.find_element(By.XPATH, "./td[5]").text.strip()
    updated = month_row.find_element(By.XPATH, "./td[10]").text.strip()
    dane.append({
        "DataPL": current_month,
        "GO": price,
        "Updated": updated
    })
    
    # 获取当前月份行之后的同组数据行（无月份文本的行）
    sibling_rows = month_row.find_elements(By.XPATH, "./following-sibling::tr[not(./td[1]/div/span/b)]")
    for row in sibling_rows:
        # 检查是否已经到下一个有月份的行，是的话停止遍历
        next_month_row = row.find_elements(By.XPATH, "./following-sibling::tr[./td[1]/div/span/b]")
        if next_month_row:
            break
        price = row.find_element(By.XPATH, "./td[5]").text.strip()
        updated = row.find_element(By.XPATH, "./td[10]").text.strip()
        dane.append({
            "DataPL": current_month,
            "GO": price,
            "Updated": updated
        })

df = pd.DataFrame(dane)

内容的提问来源于stack exchange，提问作者Adii