动态网站房贷数据爬取求助：BeautifulSoup受限，Selenium遇阻

需求与目标 #

我是Web Scraping新手，需要从https://www.canstar.com.au/home-loans/爬取房贷列表数据，目标提取以下格式的条目：

• Homestar Finance | Star Essentials P&I 80% | Variable
• Unloan | Home Loan LVR <80% | Variable
• TicToc Home Loans | Live-in Variable P&I | Variable
• ubank | Neat Home Loan Owner Occupied P&I 70-80% | Variable
  并整理为嵌套列表格式：results = [[Homestar Finance, Star Essentials P&I 80%, Variable], ...]

现有尝试与问题 #

1. BeautifulSoup离线爬取可行，但在线被反爬拦截
   离线页面爬取代码能实现目标效果，但访问真实网站时因重复请求被反爬拦截：

   import pandas as pd
   from bs4 import BeautifulSoup
   
   with open('/local/path/canstar.html', 'r') as canstar_offline :
       content = canstar_offline.read()
   
   results = [['Affiliate', 'Product Name', 'Product Type']]
       
   soup = BeautifulSoup(content, 'lxml')
   
   for listing in soup.find_all('div', class_='table-cards-container') :
       for listing1 in listing.find_all('a') :
           if listing1.text.strip() != 'More details' and listing1.text.strip() != '' :
               results.append(listing1.text.strip().split(' | '))
      
   df = pd.DataFrame(results[1:], columns=results[0]).to_dict('list')
   df2 = pd.DataFrame(df)
   
   print(df2)
   

2. Selenium尝试遇到瓶颈

   • 第一段代码仅返回一条列表数据：

     import time
     from selenium.webdriver.common.by import By
     
     url = 'https://www.canstar.com.au/home-loans'
     
     results = []
     
     driver = webdriver.Chrome()
     driver.get(url)
     
     time.sleep(3)
     tables = driver.find_elements(By.CLASS_NAME, 'table-cards-container')
     for table in tables :
         listing = table.find_element(By.TAG_NAME, 'a')
         print(listing.text)
     

   • 第二段代码提取到table-cards-container类下的全部文本，但无法筛选出目标条目：

     import time
     from selenium.webdriver.common.by import By
     
     url = 'https://www.canstar.com.au/home-loans'
     
     results = []
     
     driver = webdriver.Chrome()
     driver.get(url)
     
     time.sleep(3)
     tables = driver.find_elements(By.CLASS_NAME, 'table-cards-container')
     for table in tables :
         print(table.text)
     

解决建议与改进代码 #

核心改进点 #

• 替换固定等待为显式等待，避免页面加载慢导致元素未找到
• 精准定位目标链接，排除"More details"按钮，只提取房贷信息文本
• 模拟滚动页面加载更多动态内容（该网站需滚动触发完整列表加载）
• 配置浏览器参数，模拟真实用户行为规避反爬

完整改进代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

url = 'https://www.canstar.com.au/home-loans'
results = [['Affiliate', 'Product Name', 'Product Type']]

# 配置浏览器参数，模拟真实用户
options = webdriver.ChromeOptions()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
options.add_argument("--disable-blink-features=AutomationControlled")  # 绕过自动化检测

driver = webdriver.Chrome(options=options)
driver.get(url)

try:
    # 等待目标容器加载完成
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'table-cards-container'))
    )

    # 模拟滚动加载更多内容（滚动次数可根据需求调整）
    for _ in range(3):
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(2)  # 等待滚动后内容加载

    # 获取所有目标容器
    tables = driver.find_elements(By.CLASS_NAME, 'table-cards-container')

    for table in tables:
        # 找到容器内所有a标签，排除"More details"
        links = table.find_elements(By.TAG_NAME, 'a')
        for link in links:
            text = link.text.strip()
            if text and text != 'More details':
                # 分割文本并添加到结果列表
                parts = text.split(' | ')
                if len(parts) == 3:  # 确保格式符合要求
                    results.append(parts)

finally:
    driver.quit()

# 输出结果
print("提取的房贷数据：")
for item in results[1:]:
    print(item)

# 可选：转为DataFrame格式
import pandas as pd
df = pd.DataFrame(results[1:], columns=results[0])
print("\nDataFrame格式：")
print(df)

代码说明 #

1. 浏览器配置：通过ChromeOptions设置真实User-Agent并禁用自动化检测，降低被反爬识别的概率
2. 显式等待：等待目标元素加载完成，比固定time.sleep更稳定可靠
3. 滚动加载：该网站房贷列表为动态加载，需滚动页面触发更多内容加载
4. 精准筛选：遍历容器内所有a标签，排除无用的"More details"按钮，只保留符合格式的房贷信息
5. 格式校验：确保分割后的文本包含3个字段，避免异常数据混入结果

内容的提问来源于stack exchange，提问作者notoriousBBG