使用Selenium获取URL打印异常，输出为WebElement对象

我尝试用Selenium提取Airbnb搜索页面中的房间链接，执行了以下代码：

links = driver.find_elements(By.CSS_SELECTOR, "meta[content*='www.airbnb.com.au/rooms/']")
print(links)

但打印结果并非目标URL，而是WebElement对象实例，示例输出如下：

[<selenium.webdriver.remote.webelement.WebElement (session="faf70ce53ba59d6f6995883b0edfc006", element="f.CD9BA18B85DC3350F27BC5BF71FF60B2.d.5C798985F0B6BFD4214898A37DFC2A30.e.81")>, <selenium.webdriver.remote.webelement.WebElement (session="faf70ce53ba59d6f6995883b0edfc006", element="f.CD9BA18B85DC3350F27BC5BF71FF60B2.d.5C798985F0B6BFD4214898A37DFC2A30.e.82")>, <selenium.webdriver.remote.webelement.WebElement (session="faf70ce53ba59d6f6995883b0edfc006", element="f.CD9BA18B85DC3350F27BC5BF71FF60B2.d.5C798985F0B6BFD4214898A37DFC2A30.e.83")>, <selenium.webdriver.remote.webelement.WebElement (session="faf70ce53ba59d6f6995883b0edfc006", element="f.CD9BA18B85DC3350F27BC5BF71FF60B2.d.5C798985F0B6BFD4214898A37DFC2A30.e.84")>]

目标爬取页面为Airbnb澳大利亚站的黄金海岸房源搜索页。

解决方法 #

• 提取meta标签的content属性：find_elements返回的是WebElement对象列表，需遍历每个元素，通过get_attribute('content')获取URL：

links = driver.find_elements(By.CSS_SELECTOR, "meta[content*='www.airbnb.com.au/rooms/']")
room_urls = [link.get_attribute('content') for link in links]
print(room_urls)

• 直接抓取房源卡片链接：页面内的房源卡片通常包含直接指向房间页的<a>标签，可使用更精准的选择器，同时去重：

room_cards = driver.find_elements(By.CSS_SELECTOR, "a[href*='/rooms/']")
room_urls = [card.get_attribute('href') for card in room_cards]
# 去重处理
room_urls = list(set(room_urls))

• 处理动态加载：Airbnb页面为动态渲染，需等待元素加载完成，使用显式等待确保元素存在：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "meta[content*='www.airbnb.com.au/rooms/']")))
room_urls = [link.get_attribute('content') for link in links]
print(room_urls)

内容的提问来源于stack exchange，提问作者Clay Burnett