解决网页爬虫中TypeError: 'NoneType' object is not callable错误

问题场景 #

尝试编写Python脚本访问CoinGecko新加密货币页面，遍历所有币种、进入详情页检查Telegram链接，但执行element.click()时触发TypeError: 'NoneType' object is not callable错误。

原代码：

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

# Set up a webdriver to simulate a browser
driver = webdriver.Chrome()

# Send a request to the webpage
response = requests.get('https://www.coingecko.com/en/new-cryptocurrencies')

# Parse the response
soup = BeautifulSoup(response.text, 'html.parser')

# Find all elements with the class "coin-name"
coin_elements = soup.find_all(class_='coin-name')

# Iterate through the coin elements
for element in coin_elements:
  # Extract the text from the element (the coin name)
  coin_name = element.text
  
  # Simulate a click on the element
  element.click()
  
  # Wait for the page to load
  time.sleep(2)
  
  # Parse the content of the coin's page
  coin_soup = BeautifulSoup(driver.page_source, 'html.parser')
  
  # Find the element containing the Telegram group link
  telegram_element = coin_soup.find('a', href=re.compile(r'^https://t\.me/'))
  
  # Check if the element was found
  if telegram_element is not None:
    # Extract the link from the element
    telegram_link = telegram_element['href']
    
    # Print the coin name and link
    print(f'{coin_name}: {telegram_link}')
  else:
    # Print a message if the element was not found
    print(f'{coin_name}: No Telegram group found')

# Close the webdriver
driver.close()

错误信息：

line 23, in <module>
    element.click()
TypeError: 'NoneType' object is not callable

错误原因 #

1. 混淆页面处理逻辑：用requests+BeautifulSoup解析得到的element是BeautifulSoup的Tag对象，并非Selenium可交互的WebElement，根本没有click()方法；同时初始化的Selenium浏览器实例driver未加载目标页面，和requests请求的页面是完全独立的两个环境。
2. 缺失依赖模块：代码使用了time.sleep()和re.compile()，但未导入time、re模块。
3. 元素定位逻辑错误：即便coin_elements能找到元素，也无法通过BeautifulSoup的Tag执行点击操作。

修复方案 #

统一用Selenium控制浏览器完成所有操作，结合显式等待提升稳定性，核心调整：

• 用Selenium加载目标页面，摒弃requests+BeautifulSoup的混合逻辑
• 用Selenium定位方法获取可交互的币种元素
• 先收集所有币种的名称和详情页URL，避免页面跳转后元素失效
• 导入缺失的依赖模块
• 用显式等待替代硬编码sleep，提升脚本可靠性

修复后完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import re

# 初始化浏览器驱动
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

try:
    # 加载目标页面
    driver.get('https://www.coingecko.com/en/new-cryptocurrencies')
    
    # 等待币种元素加载完成，获取所有币种名称元素
    coin_elements = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'coin-name')))
    
    # 先收集所有币种的名称和详情页链接，避免页面跳转后原元素失效
    coin_list = []
    for elem in coin_elements:
        coin_name = elem.text.strip()
        # 找到包裹coin-name的父级a标签（实际跳转链接）
        link_elem = elem.find_element(By.XPATH, './ancestor::a')
        coin_url = link_elem.get_attribute('href')
        coin_list.append((coin_name, coin_url))
    
    # 遍历每个币种的详情页
    for coin_name, coin_url in coin_list:
        driver.get(coin_url)
        time.sleep(2)  # 等待页面加载
        
        # 查找Telegram链接
        telegram_link = None
        try:
            telegram_elem = driver.find_element(By.XPATH, '//a[starts-with(@href, "https://t.me/")]')
            telegram_link = telegram_elem.get_attribute('href')
        except:
            pass
        
        # 输出结果
        if telegram_link:
            print(f'{coin_name}: {telegram_link}')
        else:
            print(f'{coin_name}: 未找到Telegram链接')

finally:
    # 关闭浏览器
    driver.quit()

关键说明 #

• 先收集币种名称和详情页URL：页面跳转后，原页面的元素会变为失效状态，提前收集可避免报错
• 显式等待WebDriverWait：确保元素加载完成后再执行操作，比固定时长的sleep更适配不同网络环境
• 父级a标签定位：coin-name本身是span标签，无跳转功能，实际跳转链接在其祖先a标签中

内容的提问来源于stack exchange，提问作者dariocodes