Selenium+Python爬取Merkurbets赛事数据失败求助

疑问与背景 #

我尝试用Python+Selenium爬取Merkurbets网站的赛事及赔率数据，遇到一个疑问：带有team-name类的元素如下，其中div标签的_ngcontent-ng-c1043474636属性是否会影响Selenium获取该div的文本内容？

<div _ngcontent-ng-c1043474636="" class="team-name">FAC Wien</div>

尝试的定位方法示例 #

示例1：通过完整XPATH定位单场赛事元素 #

我尝试过通过完整XPATH定位赛事元素并提取数据，但返回空列表：

# games_elements = driver.find_elements(By.XPATH, '//*[@id="cdk-accordion-child-3"]/div/div/games-list-partial/div')
games_elements = driver.find_elements(By.XPATH,'//*[@id="cdk-accordion-child-3"]/div/div/games-list-partial/div/games-list-game[1]')
print('games_elements: ', games_elements)
for game in games_elements:

    try:
        # 提取球队名称
        teams = game.find_elements(By.XPATH, './/*[@id="cdk-accordion-child-3"]/div/div/games-list-partial/div/games-list-game[1]/div/div[1]/div/game-details-summary/div/div/div/div[1]')
        # 提取赔率
        odds = game.find_elements(By.XPATH, './/*[@id="cdk-accordion-child-3"]/div/div/games-list-partial/div/games-list-game[1]/div/div[2]/div[1]/market/div/div[2]/div/tip[1]/div/button/span[2]/div/tip-odds/div/div')

        if len(teams) == 2 and len(odds) >= 3:
            home_team = teams[0].text
            away_team = teams[1].text
            home_odds = odds[0].text
            draw_odds = odds[1].text
            away_odds = odds[2].text

            games_data.append({
                "Home Team": home_team,
                "Away Team": away_team,
                "Home Odds": home_odds,
                "Draw Odds": draw_odds,
                "Away Odds": away_odds,
                "Zeitstempel": timestamp  # 时间戳
            })
        else:
            logging.warning("发现异常数量的球队或赔率。")
    except NoSuchElementException as e:
        logging.error(f"提取单场赛事数据出错: {e}")

示例2：使用显式等待定位team-name元素 #

我也尝试过用显式等待等待元素可见，但依然无法获取：

# 备选方案：尝试更长等待时间并检查可见性
wait = WebDriverWait(driver, 30)
try:
    # 获取可见的'team-name'元素
    team_name_element = wait.until(
        EC.visibility_of_element_located((By.CLASS_NAME, "team-name"))
    )
    print(team_name_element.text)
except Exception as e:
    print(f"错误: {e}")
finally:
    # 保存页面内容用于检查
    page_source = driver.page_source
    with open('page_source.html', 'w', encoding='utf-8') as f:
        f.write(page_source)
    print("HTML内容已保存。")

示例3：完整流程（含Cookie接受与滚动） #

每次爬取都会先接受Cookie并滚动页面到底部，但还是无法定位到元素：

def scroll(driver):
    height = driver.execute_script("return document.body.scrollHeight")
    print('height: ', height)
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
        time.sleep(1) # <- 传统方法
        hnow = driver.execute_script("return document.body.scrollHeight")
        print('hnow: ', hnow)
        if hnow == height:
            break
        height = hnow

# 启动WebDriver
chrome_options = Options()
# chrome_options.add_argument("--headless")  # 可选：不显示窗口
service = Service(executable_path="/usr/bin/chromedriver")  # chromedriver路径
driver = webdriver.Chrome(service=service, options=chrome_options)

# 打开网页
driver.get("https://www.merkurbets.de/de/sports/0/s1/1")
driver.maximize_window()
time.sleep(5)
# 接受Cookie
accept_button = driver.find_element(By.XPATH, '//*[@id="cookiescript_accept"]')
accept_button.click()
time.sleep(2)
scroll(driver)
# 等待10秒（仅用于调试）
time.sleep(2)
team = driver.find_elements(By.XPATH,  './/div[contains(@class,"team-name")]')

求助 #

我尝试了XPATH、CSS_SELECTOR、CLASS_NAME等多种定位方式，但每次都得到空列表，无法获取目标元素，恳请提供解决思路。

内容的提问来源于stack exchange，提问作者sticki