使用BeautifulSoup爬取分类链接时遇TypeError错误求助

问题原因 #

你在循环内部的try块中，将函数名get_sub_category_links重新赋值为字符串类型的子分类链接，导致后续循环调用该函数时，这个名字已经不再指向原函数，而是一个字符串，因此触发str object is not callable错误。

具体错误行：

get_sub_category_links = subcat_obj

修复方案 #

将赋值的变量名改为和函数名不重复的名称，避免覆盖函数引用。

修改后的完整代码 #

import requests
from tqdm import tqdm
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import *
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import json
import pandas as pd
from unidecode import unidecode
from webdriver_manager.chrome import ChromeDriverManager
browser = webdriver.Chrome(ChromeDriverManager().install())

URL = 'https://www.tapwarehouse.com/'

def get_category_links(URL):
    category_links = []
    browser.get(URL)
    html = browser.page_source
    soup = BeautifulSoup(html,'html5lib')
    cat=soup.find_all("ul",{"class":"c-nav__list"})[0].find_all('a')
    for i in cat:
        try:
            link=i["href"]
            if link=='javascript:void(0)':
                pass
            else:
                category_links.append("https://www.tapwarehouse.com"+i["href"])
        except:
            pass
    return category_links

def get_sub_category_links(URL):
    sub_category_links=[]
    browser.get(URL)
    html = browser.page_source
    soup = BeautifulSoup(html,'html5lib')
    for link in soup.find_all('a', {'class': "m-categories__menu__link"}):
        sub_category_links.append("https://www.tapwarehouse.com/"+link["href"])
    return sub_category_links

response = []
sublist=[]
urllist=[]
for cat_link in get_category_links(URL = URL):
    for subcat_obj in get_sub_category_links(URL = cat_link):
        try:
            # 修改变量名，避免覆盖函数
            subcat_link = subcat_obj
            print(f'sub category is {subcat_link}')
            sublist.append(subcat_link)
            sublist = list(set(sublist))
        except:
            pass

内容的提问来源于stack exchange，提问作者crawlers