使用Selenium Python分页爬取时元素不可点击问题求助

核心问题原因 #

1. 元素被遮挡：点击下一页按钮时，顶部固定导航栏可能覆盖按钮位置，导致点击被拦截
2. 未等待页面渲染：翻页后直接执行爬取，新页面元素还未加载完成，引发元素查找失败
3. 爬取逻辑漏洞：服务类型处理中使用exit会直接终止程序，数据未正确存入DataFrame

修复方案 #

1. 用显式等待替代强制sleep #

确保元素完全加载后再操作，比time.sleep()更可靠且高效

2. 滚动到目标元素并处理遮挡 #

通过滚动让按钮进入可视区域，或用JavaScript执行点击绕过遮挡问题

3. 修正数据收集逻辑 #

修复服务类型处理的错误，将爬取结果存入DataFrame并导出

修改后的完整代码 #

import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def page_scrape(driver, wait, results_df):
    # 等待页面上的店铺列表加载完成
    ADDRESS_LOCATIONS_TEASER = '//div[contains(@class,"all-stores accordian ng-star-inserted")]'
    teaser = wait.until(EC.presence_of_element_located((By.XPATH, ADDRESS_LOCATIONS_TEASER)))
    
    ADDRESS_LOCATIONS = './/div[contains(@class,"accordian-header")]'
    locations = teaser.find_elements(By.XPATH, ADDRESS_LOCATIONS)

    for loc in locations:
        # 点击店铺展开详情
        loc.find_element(By.XPATH, './/a[@href]').click()
        
        # 等待地址元素加载完成
        add1 = wait.until(EC.presence_of_element_located((By.XPATH, ".//address[contains(@class, 'address-block')]"))).text
        address_lines = add1.split('\n')
        full_address = address_lines[0] if len(address_lines) > 0 else ''
        plz_city = address_lines[1] if len(address_lines) > 1 else ''
        telephone = address_lines[2] if len(address_lines) > 2 else ''

        # 收集服务类型
        types_of_services = '//div[contains(@id,"divrestaurant2")]'
        types_container = wait.until(EC.presence_of_element_located((By.XPATH, types_of_services)))
        types_of_service_list = types_container.find_elements(By.XPATH, './/div[contains(@class,"services ng-star-inserted")]')
        
        service_text = []
        for types in types_of_service_list:
            text = types.text.strip()
            if text == 'Services in diesem Restaurant':
                continue  # 跳过标题行
            elif text == 'Betreiber':
                break  # 遇到Betreiber时停止收集服务
            else:
                service_text.append(text)
        services = ', '.join(service_text)

        # 收集餐厅服务选项（堂食/外带等）
        Services_Types_TEASER = '//div[contains(@class, "store-accordian store-accordian-flex ng-star-inserted")]'
        services_teaser = wait.until(EC.presence_of_element_located((By.XPATH, Services_Types_TEASER)))
        services_list = services_teaser.find_elements(By.XPATH, './/div[contains(@class, "store-dine-flx ng-star-inserted")]')
        dining_options = ', '.join([s.text.strip() for s in services_list])

        # 将数据添加到DataFrame
        results_df.loc[len(results_df)] = {
            'address': full_address,
            'PLZ': plz_city.split(' ')[0] if plz_city else '',
            'Telephone': telephone,
            'Restaurant Services': f"{services} | {dining_options}"
        }

        print(f"已收集店铺信息：{full_address}")

# 初始化浏览器和等待对象
driver = webdriver.Chrome("C:/Users/doyel/Downloads/chromedriver_win32/chromedriver.exe")
driver.maximize_window()
wait = WebDriverWait(driver, 10)
driver.get('https://www.kfc.de/find-a-kfc')

# 初始化结果DataFrame
results = pd.DataFrame(columns=['address', 'PLZ', 'Telephone', 'Restaurant Services'])

# 处理Cookie弹窗
COOKIE_PATH = '//button[contains(@id,"onetrust-accept-btn-handler")]'
wait.until(EC.element_to_be_clickable((By.XPATH, COOKIE_PATH))).click()

# 翻页爬取逻辑
while True:
    page_scrape(driver, wait, results)
    
    # 处理下一页
    next_page_xpath = '//a[@aria-label="Next page"]'
    try:
        next_btn = wait.until(EC.presence_of_element_located((By.XPATH, next_page_xpath)))
        # 滚动到按钮位置，避免被导航栏遮挡
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", next_btn)
        # 用JavaScript点击，绕过遮挡问题
        driver.execute_script("arguments[0].click();", next_btn)
        print("已跳转至下一页")
        # 等待页面刷新完成
        wait.until(EC.staleness_of(driver.find_element(By.XPATH, '//div[contains(@class,"all-stores accordian ng-star-inserted")]')))
    except:
        print("已爬取至最后一页")
        break

# 保存结果到CSV
results.to_csv('kfc_germany_stores.csv', index=False, encoding='utf-8-sig')
print(f"爬取完成，共收集{len(results)}条店铺数据，已保存到kfc_germany_stores.csv")

driver.quit()

关键优化点说明 #

• 所有元素操作前均添加显式等待，确保元素加载完成
• 使用JavaScript点击下一页按钮，彻底解决元素遮挡问题
• 修复服务类型收集逻辑，正确过滤无关内容并拼接信息
• 将爬取数据结构化存入DataFrame，最终导出为CSV文件
• 翻页后等待原页面元素失效，确保新页面完全加载

内容的提问来源于stack exchange，提问作者Doyel De Sarkar