使用BeautifulSoup/Selenium爬取Clutch.co数据遇阻，求解决方案

一、初始代码NameError错误解决 #

错误原因 #

urljoin 函数属于 urllib.parse 模块，代码中未导入该模块，导致运行时触发NameError: name 'urljoin' is not defined。

解决步骤 #

在代码开头添加模块导入语句：

from urllib.parse import urljoin

修改后的完整代码 #

import requests
from bs4 import BeautifulSoup
import time
from urllib.parse import urljoin  # 新增导入

def scrape_clutch_digital_agencies(url):
    # Set a User-Agent header
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
        'Referer': 'https://clutch.co/',
        'Accept-Language': 'en-US,en;q=0.9'
    }

    # Create a session to handle cookies
    session = requests.Session()

    # Check the robots.txt file
    robots_url = urljoin(url, '/robots.txt')
    robots_response = session.get(robots_url, headers=headers)

    # Print robots.txt content (for informational purposes)
    print("Robots.txt content:")
    print(robots_response.text)

    # Wait for a random interval before making the first request
    time.sleep(time.uniform(2, 4))

    # Send an HTTP request to the URL
    response = session.get(url, headers=headers)

    # Check if the request was successful (status code 200)
    if response.status_code == 200:
        # Parse the HTML content of the page
        soup = BeautifulSoup(response.text, 'html.parser')

        # Find the elements containing agency names (adjust this based on the website structure)
        agency_name_elements = soup.select('.company-info .company-name')

        # Extract and print the agency names
        agency_names = [element.get_text(strip=True) for element in agency_name_elements]

        print("Digital Agencies in Israel:")
        for name in agency_names:
            print(name)
    else:
        print(f"Failed to retrieve the page. Status code: {response.status_code}")

# Example usage
url = 'https://clutch.co/il/agencies/digital'
scrape_clutch_digital_agencies(url)

二、反爬机制应对方案 #

1. Requests + BeautifulSoup 深度优化 #

Clutch.co的反爬主要检测请求特征、访问频率和IP，可通过以下方式突破：

• 完善请求头：补充Referer、Accept-Language等字段，模拟真实浏览器请求
• 随机请求间隔：使用random.uniform()生成2-4秒的随机延迟，避免机械固定间隔
• 代理IP轮换：使用代理池（免费或付费代理服务），每次请求切换IP，防止IP被封禁
• 会话保持：使用requests.Session()维持Cookie，模拟用户持续访问的状态
• 遵守robots规则：先通过robots.txt确认允许爬取的路径，避免触碰禁止区域

2. Selenium 反爬突破方案 #

Selenium被检测的核心原因是浏览器指纹暴露自动化特征，优化方案如下：

• 使用undetected-chromedriver：该驱动专门绕过反爬检测，隐藏Selenium的自动化痕迹
• 伪装浏览器参数：添加禁用自动化检测、模拟真实设备的参数
• 显式等待替代固定休眠：使用WebDriverWait等待元素加载，避免固定延迟的机械感
• 模拟人类行为：添加鼠标移动、页面滚动等操作，让爬取行为更接近真实用户
• 配置代理IP：给Chrome浏览器配置代理，避免IP被限制

优化后的Selenium代码（使用undetected-chromedriver）

首先安装依赖：

pip install undetected-chromedriver

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import undetected_chromedriver as uc
import random
import time

def scrape_clutch_digital_agencies_with_selenium(url):
    # 配置Chrome选项
    chrome_options = uc.ChromeOptions()
    chrome_options.add_argument('--headless=new')  # 新版无头模式，更隐蔽
    chrome_options.add_argument('--disable-blink-features=AutomationControlled')
    chrome_options.add_argument('--start-maximized')
    chrome_options.add_argument('lang=en-US')

    # 初始化undetected-chromedriver
    driver = uc.Chrome(options=chrome_options)

    try:
        driver.get(url)

        # 显式等待目标元素加载，替代固定sleep
        WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, '.company-info .company-name'))
        )

        # 模拟人类滚动页面
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(random.uniform(1, 3))

        # 获取页面源码
        page_source = driver.page_source
        soup = BeautifulSoup(page_source, 'html.parser')

        # 提取代理机构名称
        agency_name_elements = soup.select('.company-info .company-name')
        agency_names = [element.get_text(strip=True) for element in agency_name_elements]

        print("Digital Agencies in Israel:")
        for name in agency_names:
            print(name)
    finally:
        # 确保浏览器关闭
        driver.quit()

# Example usage
url = 'https://clutch.co/il/agencies/digital'
scrape_clutch_digital_agencies_with_selenium(url)

内容的提问来源于stack exchange，提问作者zero