如何使用urllib爬取谷歌搜索结果不被封禁,批量获取目标URL
核心优化方案(优先降低请求量)
谷歌搜索支持num URL参数控制单页返回结果数量,最高可设置为100条/页。你要爬1000条结果仅需发送10次请求,相比默认10条/页的请求量减少90%,从根源上降低触发风控的概率。
现有代码问题&修改点
- 你定义的UA池没有被实际调用,每次请求都用固定的
Mozilla/5.0标识,特征过于明显 - 请求头仅带User-Agent,缺少正常浏览器必带的Accept、Accept-Language等字段,极易被识别为爬虫
- 每次爬取单条结果就开关一次CSV文件,冗余IO操作无意义
- 没有携带请求返回的Cookie,不符合真实用户的访问行为
修改后可运行代码
from urllib.request import Request, urlopen, HTTPCookieProcessor, build_opener from bs4 import BeautifulSoup import random import time from urllib.error import HTTPError user_agent_list = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Edg/90.0.818.56', 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko' ] # 启用Cookie处理,模拟浏览器行为 cookie_handler = HTTPCookieProcessor() opener = build_opener(cookie_handler) root = "https://www.google.com/" # 加入num=100参数,单页返回100条结果 base_url = "https://google.com/search?q={}&num=100" csv_fn = 'export.csv' # 提前打开文件,避免重复IO document = open(csv_fn, "a", encoding='utf-8') def news(link, n_pages=10, page_count=0): print(f"正在爬取第{page_count+1}页") # 每爬3页插入一次长停顿,模拟用户休息 if page_count %3 ==0 and page_count !=0: sleep_time = random.uniform(60, 120) print(f"触发长停顿,等待{int(sleep_time)}秒") time.sleep(sleep_time) else: time.sleep(random.uniform(20, 40)) # 随机取UA headers = { "User-Agent": random.choice(user_agent_list), "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2", "Referer": "https://www.google.com/" } req = Request(link, headers=headers) try: webpage = opener.open(req).read() except HTTPError as e: if e.code == 429 or e.code ==503: print("触发谷歌限流,等待30分钟后重试") time.sleep(1800) news(link, n_pages=n_pages, page_count=page_count) return else: raise e soup = BeautifulSoup(webpage, 'html5lib') for item in soup.find_all('div', attrs = {'class': 'ZINbbc xpd O9g5cc uUPGi'}): try: title = (item.find('div', attrs= {'class': 'BNeawe vvjwJb AP7Wnd'}).get_text()) # 批量写入文件 document.write(f"{title}\n") except: print('No title found') continue nextPage = soup.find('a', attrs = {'aria-label': "Volgende pagina"}) if nextPage is None: nextPage = soup.find('a', attrs = {'aria-label': "Next"}) page_count += 1 if nextPage is not None and page_count < n_pages: nextPage = (nextPage['href']) link = root + nextPage news(link, n_pages=n_pages, page_count=page_count) elif page_count >= n_pages: print(f'累计爬取{n_pages}页,约{n_pages*100}条结果,任务完成') document.close() else: print('没有下一页,爬取终止') document.close() # 构造初始搜索链接,单页100条 keyword = 'whale+watching+orca+iceland' link = base_url.format(keyword) # 10页刚好1000条左右,总耗时约3小时符合要求 news(link, n_pages = 10)
额外防封禁实用建议
- 不要连续爬取同一个关键词的结果,可以穿插2-3个无关关键词的搜索请求,再回到目标关键词爬取,模拟正常用户的搜索行为
- 如果有多个关键词需要爬取,打乱关键词的爬取顺序,不要按照固定顺序逐个爬取
- 不要在公共网络下爬取,尽量用家庭宽带IP,同IP下不要有其他高频访问谷歌的行为
- 如果触发验证码页面,不要继续发送请求,手动在浏览器里完成一次谷歌搜索和验证码验证,等待1-2小时后再继续爬取即可
内容的提问来源于stack exchange,提问作者CrossLord
相关产品推荐
相关产品推荐

