混合使用Selenium与BeautifulSoup爬取是否合规？如何优化爬速？

背景与现有代码 #

我需要爬取一个包含A到Z报纸列表的网站，先获取所有报纸的URL，再从每个URL提取所有者、邮寄地址、邮箱等信息，目标网站为https://media.info/newspapers/titles。

爬取所有报纸URL的代码 #

driver.get('https://media.info/newspapers/titles')
time.sleep(2)
page_title = []
pages = driver.find_elements(By.XPATH,"//div[@class='pages']//a")
for i in pages:
    page_title.append(i.get_attribute("href"))

names = []
for i in page_title:
    driver.get(i)
    time.sleep(1)
    
    name = driver.find_elements(By.XPATH,"//div[@class='info thumbBlock']//a")
    for i in name:
        names.append(i.get_attribute("href"))

执行后得到的URL总数为1688，前5个URL如下：

names[0:5]
['https://media.info/newspapers/titles/abergavenny-chronicle',
 'https://media.info/newspapers/titles/abergavenny-free-press',
 'https://media.info/newspapers/titles/abergavenny-gazette-diary',
 'https://media.info/newspapers/titles/the-abingdon-herald',
 'https://media.info/newspapers/titles/academies-week']

提取报纸信息的代码 #

test = []
c = 0
for i in names:
    driver.get(i)
    time.sleep(2)
    
    r = requests.get(i)
    soup = BeautifulSoup(r.content,'lxml')
    
    try:
        name = driver.find_element(By.XPATH,"//*[@id='mainpage']/article/div[3]/h1").text

        try:
            twitter = driver.find_element(By.XPATH,"//*[@id='mainpage']/article/table[3]/tbody/tr/td[1]/a").text
        except:
            twitter = None

        try:
            twitter_followers = driver.find_element(By.XPATH,"//*[@id='mainpage']/article/table[3]/tbody/tr/td[1]/small").text.replace(' followers','').lstrip('(').rstrip(')')
        except:
            twitter_followers = None
            
        people = []
        try:
            persons = driver.find_elements(By.XPATH,"//div[@class='columns']")
            for i in persons:
                people.append(i.text)
        except:
            people.append(None)

        try:
            owner = soup.select_one('th:contains("Owner") + td').text
        except:
            owner = None

        try:
            postal_address = soup.select_one('th:contains("Postal address") + td').text
        except:
            postal_address = None

        try:
            Telephone = soup.select_one('th:contains("Telephone") + td').text
        except:
            Telephone = None

        try:
            company_website = soup.select_one('th:contains("Official website") + td > a').get('href')
        except:
            company_website = None

        try:
            main_email = soup.select_one('th:contains("Main email") + td').text
        except:
            main_email = None

        try:
            personal_email = soup.select_one('th:contains("Personal email") + td').text
        except:
            personal_email = None

        r2 = requests.get(company_website)
        soup2 = BeautifulSoup(r2.content,'lxml')

        try:
            is_wordpress = soup2.find("meta",{"name":"generator"}).get('content')
        except:
            is_wordpress = None

        news_Data = {
                    "Name": name,
                    "Owner": owner,
                    "Postal Address": postal_address,
                    "main Email":main_email,
                    "Telephone": Telephone, 
                    "Personal Email": personal_email,
                    "Company Wesbite": company_website,
                    "Twitter_Handle": twitter,
                    "Twitter_Followers": twitter_followers,
                    "People":people,
                    "Is Wordpress?":is_wordpress
                    }

        test.append(news_Data)
        c=c+1
        print("completed",c)

    except Exception as Argument:
        print(f"There is an exception with {i}")
        pass

疑问与解答 #

1. 同时使用Selenium和BeautifulSoup是否属于良好的开发实践？ #

这不是最优实践，核心问题在于你当前的代码存在重复请求：同一个页面既用Selenium的driver.get加载，又用requests.get重新请求，完全浪费了资源和时间。

合理的混用场景应该是：仅在页面需要动态渲染（比如JS加载内容）时用Selenium获取页面源码，然后传给BeautifulSoup解析；如果页面是纯静态的，直接用requests+BeautifulSoup即可，速度比Selenium快得多。避免无意义的重复请求，两种工具应该互补而非重复使用。

2. 当前代码运行耗时过长，有哪些替代方案可以缩短代码的运行时间？ #

针对你的代码，可从以下几个方向优化：

• 消除重复请求：同一个页面不要同时用Selenium和requests请求。比如用Selenium加载页面后，直接用driver.page_source获取源码传给BeautifulSoup：

  soup = BeautifulSoup(driver.page_source, 'lxml')
  

  省掉一次requests.get的时间。
• 替换固定等待为显式等待：time.sleep是固定时长等待，不管页面是否加载完成。改用Selenium的WebDriverWait等待目标元素出现，减少不必要的等待：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  # 等待标题元素出现，最长等10秒
  name_element = WebDriverWait(driver, 10).until(
      EC.presence_of_element_located((By.XPATH, "//*[@id='mainpage']/article/div[3]/h1"))
  )
  name = name_element.text
  

• 启用并发请求：用多线程（如threading）或异步请求（如aiohttp）批量处理报纸页面。注意控制请求频率，避免触发网站反爬机制。
• 优化Selenium配置：启用无头模式、关闭图片加载、禁用不必要的JS，减少浏览器加载开销：

  from selenium.webdriver.chrome.options import Options
  
  chrome_options = Options()
  chrome_options.add_argument("--headless=new")  # 无头模式
  chrome_options.add_argument("--disable-images")  # 禁用图片加载
  driver = webdriver.Chrome(options=chrome_options)
  

• 添加超时机制：对requests.get和Selenium操作添加超时设置，避免因单个页面加载过慢卡住整个流程：

  # requests添加超时
  r2 = requests.get(company_website, timeout=5)
  

• 提前过滤无效URL：先检查company_website是否为空，为空则跳过后续请求，避免无效操作。

内容的提问来源于stack exchange，提问作者Lalit Joshi