Python爬虫获取HTML与Chrome不一致，无法提取详情页数据求助

问题描述 #

需要提取房产详情页（示例URL：https://salesweb.civilview.com/Sales/SaleDetails?PropertyId=1171700135）的表格数据并生成CSV文件，但运行代码后出现AttributeError: 'NoneType' object has no attribute 'find_next'错误，且获取的HTML内容和浏览器实际显示不一致，无法定位到目标元素Approx. Upset*:。

用户尝试的代码如下：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup

service = ChromeService(executable_path=ChromeDriverManager().install())
chrome_options = ChromeOptions()
chrome_options.add_argument("--headless")  # Run Chrome in headless mode (without opening a window)
driver = webdriver.Chrome(service=service, options=chrome_options)

url = "https://salesweb.civilview.com/Sales/SaleDetails?PropertyId=1171700135"

driver.get(url)

table = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, "table-striped")))

html_content = driver.page_source
print("HTML is", html_content)

driver.quit()

soup = BeautifulSoup(html_content, 'html.parser')

row = soup.find('tr', string='Approx. Upset*:')

value = row.find_next('td').get_text(strip=True)

print("Approx. Upset:", value)

问题原因 #

1. 无头模式被反爬识别：默认的--headless模式浏览器特征明显，网站会检测到爬虫身份，返回经过处理的内容，导致获取的HTML和正常浏览器打开的不一致。
2. 元素定位逻辑错误：Approx. Upset*:文本是嵌套在<td>标签内的，不是<tr>的直接文本，用soup.find('tr', string='Approx. Upset*:')根本找不到对应的行元素，自然会返回None并触发后续错误。

解决方法 #

1. 修复无头模式的反爬问题 #

给Chrome无头模式添加更多模拟正常浏览器的参数，避免被反爬机制识别：

chrome_options.add_argument("--headless=new")  # 使用新版无头模式，更接近真实浏览器
chrome_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
chrome_options.add_argument("--window-size=1920,1080")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")  # 禁用自动化特征标记

2. 修正元素定位逻辑 #

不要直接查找包含目标文本的<tr>，而是先定位到文本所在的<td>，再获取其相邻的数值单元格：

target_td = soup.find('td', string='Approx. Upset*:')
if target_td:
    value_td = target_td.find_next_sibling('td')
    if value_td:
        print("Approx. Upset:", value_td.get_text(strip=True))

3. 完整优化代码（支持提取所有数据生成CSV） #

下面是可以直接运行的完整代码，能提取详情页所有表格数据并保存为CSV文件：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import csv

# 配置浏览器选项，规避反爬
chrome_options = ChromeOptions()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
chrome_options.add_argument("--window-size=1920,1080")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")

service = ChromeService(executable_path=ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=chrome_options)

url = "https://salesweb.civilview.com/Sales/SaleDetails?PropertyId=1171700135"
driver.get(url)

# 等待所有表格元素加载完成
WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "table-striped")))

html_content = driver.page_source
driver.quit()

# 解析HTML提取数据
soup = BeautifulSoup(html_content, 'html.parser')
table = soup.find('table', class_='table-striped')
rows = table.find_all('tr')

# 整理键值对数据
csv_data = []
for row in rows:
    cols = row.find_all('td')
    if len(cols) == 2:
        field = cols[0].get_text(strip=True)
        value = cols[1].get_text(strip=True)
        csv_data.append([field, value])

# 写入CSV文件
with open('property_details.csv', 'w', newline='', encoding='utf-8') as csv_file:
    writer = csv.writer(csv_file)
    writer.writerow(['字段', '值'])
    writer.writerows(csv_data)

print("数据已成功导出到property_details.csv")

内容的提问来源于stack exchange，提问作者Esh Val