新手求助：使用Selenium爬取MorphMarket无法提取指定信息

本人是Selenium新手，尝试爬取https://www.morphmarket.com/all/c/reptiles/pythons/ball-pythons页面的球蟒相关信息，目前仅能提取部分内容，但不清楚如何提取性别、公司、位置以及会员信息。以下是我的代码：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()

driver.get("https://www.morphmarket.com/all/c/reptiles/pythons/ball-pythons")
time.sleep(5)

snakes = driver.find_elements(By.CSS_SELECTOR,"a.animalCard--avL0R")

link = snakes[0].get_attribute("href")
driver.get(snakes[0].get_attribute("href"))
time.sleep(5)
genes= driver.find_element(By.CSS_SELECTOR, "h1.animalTitle--cH6qE")
print(genes.text)
snake=driver.find_element(By.CSS_SELECTOR,"h2.animalSubTitle--mhYId")
print(snake.text)
price = driver.find_element(By.CSS_SELECTOR, "h1.salePrice--qNIIs")
print(price.text)
#sex = driver.find_element(By.TAG_NAME, "span")
Birth = driver.find_elements(By.CSS_SELECTOR,"div.labelValueContainer--z1CP3")
print(Birth[1].text)
print(Birth[3].text)
print(Birth[4].text)
print(Birth[5].text)
print(Birth[6].text)
print(Birth[7].text)
print(Birth[8].text)
print(Birth[9].text)
print(Birth[10].text)
print(Birth[11].text)

Company= driver.find_element(By.CSS_SELECTOR, "h4.title--qLioF")
print(Company.text)

Location=driver.find_element(By.CSS_SELECTOR,"p.location--TtVtP")
print(Location.text)
membership= driver.find_element(By.CSS_SELECTOR, "span")

1. 提取性别 #

性别信息存放在labelValueContainer容器组中，可通过XPath定位包含"Sex"文本的容器，直接获取对应值：

sex = driver.find_element(By.XPATH, "//div[contains(text(), 'Sex')]/following-sibling::div")
print("性别:", sex.text)

2. 提取公司名称 #

原选择器可能匹配到非目标元素，需限定在卖家信息区域内，使用更精确的CSS选择器：

company = driver.find_element(By.CSS_SELECTOR, "div.sellerInfoContainer h4.title--qLioF")
print("公司:", company.text)

3. 提取位置信息 #

同样限定在卖家信息区域，确保定位到正确的位置元素：

location = driver.find_element(By.CSS_SELECTOR, "div.sellerInfoContainer p.location--TtVtP")
print("位置:", location.text)

4. 提取会员信息 #

会员标识带有特定class（如membershipBadge），可通过CSS选择器直接定位；若class动态变化，也可用XPath模糊匹配：

# 方式1：精确class定位
membership = driver.find_element(By.CSS_SELECTOR, "span.membershipBadge")
# 方式2：模糊匹配class
membership = driver.find_element(By.XPATH, "//span[contains(@class, 'membershipBadge')]")
print("会员信息:", membership.text)

优化后完整代码 #

建议用显式等待替代time.sleep，提升爬取稳定性：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

# 进入列表页并等待卡片加载
driver.get("https://www.morphmarket.com/all/c/reptiles/pythons/ball-pythons")
snakes = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "a.animalCard--avL0R")))
# 进入第一个球蟒详情页
driver.get(snakes[0].get_attribute("href"))

# 提取基础信息
genes = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h1.animalTitle--cH6qE")))
print("基因:", genes.text)
snake_subtitle = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h2.animalSubTitle--mhYId")))
print("副标题:", snake_subtitle.text)
price = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "h1.salePrice--qNIIs")))
print("价格:", price.text)

# 提取性别
sex = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(text(), 'Sex')]/following-sibling::div")))
print("性别:", sex.text)

# 提取卖家相关信息
seller_container = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.sellerInfoContainer")))
company = seller_container.find_element(By.CSS_SELECTOR, "h4.title--qLioF")
print("公司:", company.text)
location = seller_container.find_element(By.CSS_SELECTOR, "p.location--TtVtP")
print("位置:", location.text)
membership = seller_container.find_element(By.CSS_SELECTOR, "span.membershipBadge")
print("会员等级:", membership.text)

driver.quit()

注意事项 #

• 页面class可能随网站更新动态变化，若定位失败，需重新检查元素结构并调整定位表达式。
• 频繁爬取可能触发反爬机制，建议添加合理的请求间隔，避免短时间内大量访问。

内容的提问来源于stack exchange，提问作者user22428402