使用googlesearch-python批量爬取企业URL遇429错误的解决咨询

批量获取企业官方URL时，使用googlesearch-python的代码全部返回错误：
Error searching for System73 URL: HTTP Error 429: Too Many Requests

原代码因请求频率过高、无反爬规避措施，直接触发了搜索引擎的限流机制。以下是修改后的可运行代码及关键改进说明：

修改后的代码 #

from googlesearch import search
import time
from random import uniform

companies = [
    "System73", "Bo Growth", "SDG Group", "Plexus", "Amadeus", 
    "Novatec Software Engineering España SL", "Ibermática", "Swiss RE", 
    "EY", "Oxigent", "OpenNebula Systems", "Bunge", "Deloitte", "e-Frontiers", 
    "N26", "INTEL", "ToBeIT", "Globant", "Nutanix", "Ubisoft", "We Bring", 
    "ELCA Security"
]

company_urls = {}
# 模拟浏览器请求头，避免被识别为爬虫
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

for company in companies:
    retries = 3  # 最多重试3次
    success = False
    while retries > 0 and not success:
        try:
            search_query = f"{company} official website"
            # 指定Bing搜索引擎，设置随机延迟，传入自定义UA
            search_results = search(
                search_query,
                num=1,
                stop=1,
                engine="bing",
                pause=uniform(2, 5),  # 每次请求间隔2-5秒随机值
                user_agent=headers["User-Agent"]
            )
            
            company_url = next(search_results, None)
            company_urls[company] = company_url
            print(f"Successfully fetched {company}: {company_url}")
            success = True
        except Exception as e:
            retries -= 1
            print(f"Error searching for {company} URL (retries left: {retries}): {e}")
            if retries > 0:
                # 重试前额外增加3-6秒延迟
                time.sleep(uniform(3, 6))

# 输出最终结果
print("\nFinal Results:")
for company, url in company_urls.items():
    print(f"{company}: {url}")

关键改进点 #

• 浏览器UA模拟：通过自定义请求头伪装成正常浏览器访问，降低被识别为爬虫的概率
• 随机请求间隔：使用uniform()生成2-5秒的随机延迟，配合pause参数控制搜索库的请求间隔，避免固定时间触发反爬规则
• 失败重试机制：每个企业最多重试3次，重试前增加更长延迟，提升请求成功率
• 切换搜索引擎：改用反爬限制更宽松的Bing搜索，适合批量请求场景
• 清理无效数据：移除原列表中的etc.，避免无效搜索请求

内容的提问来源于stack exchange，提问作者SolidOpt