Selenium爬取lavoro.gov.it分页循环及点击拦截问题求助

问题1：分页无限循环 #

原因 #

你使用的XPath是精确匹配class属性，而页面中pager__item--next的class可能存在空格差异（比如渲染后末尾无空格），导致每次都定位到同一个“下一页”元素；同时循环中冗余的driver.get(search_results_url)会让页面反复回到当前页，进一步加剧状态混乱，引发无限循环。

修复方案 #

• 用contains()匹配class，避免精确匹配的空格问题；
• 移除冗余的页面回退操作，直接在当前页获取下一页链接并跳转；
• 记录已访问的分页URL，彻底避免重复访问。

修改后的分页逻辑：

try:
    # 使用contains匹配class，忽略空格差异
    next_page_link = driver.find_element(By.XPATH, "//li[contains(@class, 'pager__item--next')]")
    # 检查是否为禁用状态（最后一页）
    if 'disabled' not in next_page_link.get_attribute('class'):
        search_results_url = next_page_link.get_attribute("href")
        driver.get(search_results_url)
    else:
        print("[DEBUG] 已到最后一页")
        break
except Exception as e:
    print('[DEBUG] Exception:', e)
    print('[DEBUG] break')
    break

问题2：Cookie弹窗拦截点击 #

原因 #

页面底部的欧盟Cookie合规弹窗遮挡了分页按钮，导致点击“下一页”时被弹窗元素拦截。

修复方案 #

启用并完善Cookie弹窗处理逻辑，确保弹窗加载完成后关闭，必要时用JS点击绕过遮挡问题。

修改后的弹窗处理代码（可放在scrape_page函数开头）：

# 处理Cookie弹窗
try:
    print('关闭Cookie弹窗')            
    cookie_banner = driver.find_element(By.XPATH, "//*[@id='popup-buttons']/button[1]")
    # 用JS点击避免遮挡问题
    driver.execute_script("arguments[0].click();", cookie_banner)
    time.sleep(1)
except Exception as e:
    print('Cookie弹窗处理异常:', e)

完整优化后的代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

import time
import selenium
print('Selenium:', selenium.__version__)

def scrape_page(driver, keyword):
    try:
        # 处理Cookie弹窗
        try:
            print('关闭Cookie弹窗')            
            cookie_banner = WebDriverWait(driver, 10).until(
                EC.element_to_be_clickable((By.XPATH, "//*[@id='popup-buttons']/button[1]"))
            )
            driver.execute_script("arguments[0].click();", cookie_banner)
            time.sleep(1)
        except Exception as e:
            print('Cookie弹窗处理异常:', e)
        
        elements_dt = driver.find_elements(By.XPATH, "//div[@class='row']")
        print('[DEBUG] len(elements_dt):', len(elements_dt))
        data = []

        for index, element in enumerate(elements_dt, 1):
            try:
                # 使用相对路径定位a标签，避免每次取到页面第一个链接
                article_url = element.find_element(By.XPATH, './/a[@class="search-item-link"]').get_attribute("href")
                article_title = element.text
                
                data.append({
                    'keyword': keyword,
                    'Titolo': article_title, 
                    'URL': article_url, 
                })
                
                print('[DEBUG] data:', data[-1])
            except Exception as e:
                print("元素处理错误:", e)
        
        return data

    except Exception as e:
        print("页面爬取错误:", e)
        return None

# --- 主逻辑 ---
driver = webdriver.Chrome()
driver.maximize_window()
driver.implicitly_wait(10)

start_url = "https://www.lavoro.gov.it/"
all_data = []

keywords = ['big data', 'data warehouse', 'data lake', 'data science', 'data mining', 'data privacy', 'open data',
            'data governance', 'data literacy', 'data', 'biometric', 'analisi biometrica',
            'machine learning', 'apprendimento automatico', 'algoritm', 'calcolo', 'punteggio', 'valutazione',
            'monitoraggi','predittiv', 'cloud', 'statistic', 'reti neurali', 'rete neurale', 'neural network',
            'apprendimento profondo', 'deep learning', 'ranking', 'classificazione', 'apprendimento supervisionato',
            'apprendimento non supervisionato', 'software', 'numeric', 'rango', 'ranghi', 'rank', "Elaborazione del linguaggio naturale",
            'natural language processing', 'NLP', 'graduazione', 'transformer', 'GPT', 'Bard', 'Copilot', 'Gemini', 'DALL-E',
            'automa', 'internet delle cose', 'intelligenza artificiale']

for word in keywords:
    print("主页面:", start_url)
    driver.get(start_url)

    print('搜索关键词:', word)
    searchbar = driver.find_element(By.ID, "search-input")
    searchbar.send_keys(word)
    searchbar.send_keys(Keys.ENTER)
    
    # 显式等待搜索结果加载完成
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, "//div[@class='row']"))
    )
    
    search_results_url = driver.current_url
    
    visited_urls = set()  # 记录已访问URL，避免循环
    while search_results_url not in visited_urls:
        visited_urls.add(search_results_url)
        print("正在爬取:", search_results_url)
        
        page_data = scrape_page(driver, word)
        if page_data:
            all_data.extend(page_data)

        try:
            next_page_link = WebDriverWait(driver, 10).until(
                EC.element_to_be_clickable((By.XPATH, "//li[contains(@class, 'pager__item--next')]"))
            )
            # 检查是否为最后一页
            if 'disabled' in next_page_link.get_attribute('class'):
                print("[DEBUG] 已到最后一页")
                break
            search_results_url = next_page_link.get_attribute("href")
            driver.get(search_results_url)
        except Exception as e:
            print('[DEBUG] 异常:', e)
            print('[DEBUG] 结束分页')
            break
            
driver.quit()

import pandas as pd
df = pd.DataFrame(all_data)
print(df)

input("按回车键关闭")
df.to_excel('lavoro_scrape_tot.xlsx')

额外优化点 #

1. 用WebDriverWait显式等待替代time.sleep，提升代码稳定性；
2. 爬取文章链接时使用相对路径（.//a），避免每次获取到页面第一个链接；
3. 添加visited_urls集合记录已访问的分页URL，彻底杜绝无限循环；
4. 使用driver.execute_script点击Cookie按钮，解决可能的遮挡问题。

内容的提问来源于stack exchange，提问作者Roberto Artiaco