如何让bs4爬取的Google搜索结果与浏览器搜索结果一致？

以下是针对你的问题的具体优化方案，让爬虫返回和浏览器一致的搜索结果：

核心问题与优化方向 #

• 请求头不完整：仅设置User-Agent和Accept-Language不足以完全模拟浏览器请求，Google会通过其他请求头识别爬虫
• 会话状态缺失：浏览器会保持会话Cookie，而单次requests.get请求无法维持状态
• 日期格式不规范：Google搜索的日期参数要求特定格式，输入格式错误会导致日期过滤失效
• 链接解析逻辑脆弱：手动分割链接容易遗漏或错误解析有效结果

具体优化步骤 #

1. 补全请求头
   添加浏览器常见的请求头字段，同时使用完整的Chrome User-Agent字符串，让请求更接近真实浏览器：

   headers = {
       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
       "Accept-Language": "en-US,en;q=0.9",
       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
       "Referer": "https://www.google.com/",
       "DNT": "1",
       "Connection": "keep-alive",
       "Upgrade-Insecure-Requests": "1"
   }
   

2. 使用会话保持状态
   用requests.Session()代替单次请求，自动维持Cookie和会话状态，避免被Google判定为异常请求：

   session = requests.Session()
   session.headers.update(headers)
   # 先请求Google主页获取初始Cookie
   session.get("https://www.google.com/")
   req = session.get(url, verify=True)
   

3. 规范日期格式
   Google搜索的cd_min和cd_max参数要求mm/dd/yyyy格式，需确保输入符合要求，或在代码中做格式转换：

   from datetime import datetime
   try:
       # 转换yyyy-mm-dd为mm/dd/yyyy
       start_date = datetime.strptime(start_date, "%Y-%m-%d").strftime("%m/%d/%Y")
       end_date = datetime.strptime(end_date, "%Y-%m-%d").strftime("%m/%d/%Y")
   except ValueError:
       print("日期格式错误，请输入yyyy-mm-dd或mm/dd/yyyy格式")
       exit()
   

4. 优化链接解析逻辑
   用urllib.parse解析/url开头的链接，避免手动分割的错误：

   from urllib.parse import parse_qs, urlparse
   
   for link in soup.find_all("a"):
       href = link.get("href")
       if href and href.startswith("/url"):
           parsed_href = urlparse(href)
           query_params = parse_qs(parsed_href.query)
           if "q" in query_params:
               final_link = query_params["q"][0]
               # 过滤不需要的域名和关键词
               excluded_domains = ["google.com", "linkedin.com", "youtube.com"]
               excluded_keywords = ["proxyDocument", ".pdf"]
               if (final_link.startswith("https") and 
                   not any(domain in final_link for domain in excluded_domains) and
                   not any(keyword in final_link for keyword in excluded_keywords)):
                   print(final_link)
   

5. 避免频繁请求
   在请求之间添加1-3秒的随机延迟，模拟人类浏览行为，降低被反爬拦截的概率：

   import time
   import random
   time.sleep(random.uniform(1, 3))
   

修改后的完整代码 #

from bs4 import BeautifulSoup
import requests
from ssl import SSLCertVerificationError
from urllib3.exceptions import MaxRetryError
import urllib.parse
import urllib3
from urllib.error import HTTPError, URLError
from datetime import datetime
import time
import random

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# 补全请求头
headers = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "Referer": "https://www.google.com/",
    "DNT": "1",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1"
}

start_date = input("Enter start date (yyyy-mm-dd or mm/dd/yyyy): ")
end_date = input("Enter end date (yyyy-mm-dd or mm/dd/yyyy): ")
company_name = input("Enter company name: ")
keyword = input("Enter keyword: ")

# 统一日期格式为mm/dd/yyyy
try:
    start_date = datetime.strptime(start_date, "%Y-%m-%d").strftime("%m/%d/%Y")
except ValueError:
    try:
        start_date = datetime.strptime(start_date, "%m/%d/%Y").strftime("%m/%d/%Y")
    except ValueError:
        print("Invalid start date format. Please use yyyy-mm-dd or mm/dd/yyyy.")
        exit()

try:
    end_date = datetime.strptime(end_date, "%Y-%m-%d").strftime("%m/%d/%Y")
except ValueError:
    try:
        end_date = datetime.strptime(end_date, "%m/%d/%Y").strftime("%m/%d/%Y")
    except ValueError:
        print("Invalid end date format. Please use yyyy-mm-dd or mm/dd/yyyy.")
        exit()

try:
    url_params = {
        'q': f'{company_name} company {keyword} technology ',
        'tbs': f'cdr:1,cd_min:{start_date},cd_max:{end_date}',
        'source': 'lnms',
        'tbm': 'nws'
    }

    url = 'https://www.google.com/search?' + urllib.parse.urlencode(url_params)

    # 使用会话维持状态
    session = requests.Session()
    session.headers.update(headers)
    # 先访问Google主页获取初始Cookie
    session.get("https://www.google.com/")
    # 添加随机延迟
    time.sleep(random.uniform(1, 3))
    req = session.get(url, verify=True)
    req.raise_for_status()  # 检查请求是否成功
    soup = BeautifulSoup(req.content, 'lxml')

except (requests.exceptions.SSLError, SSLCertVerificationError, MaxRetryError,
        requests.exceptions.InvalidSchema, requests.exceptions.MissingSchema,
        requests.exceptions.ConnectionError, requests.exceptions.TooManyRedirects,
        requests.exceptions.ChunkedEncodingError, TypeError, HTTPError, URLError,
        ValueError, requests.exceptions.HTTPError) as e:
    print(f'Connection failed with this website: {str(e)}')

# 优化链接解析
excluded_domains = ["google.com", "linkedin.com", "youtube.com", "nasdaq.com",
                    "quora.com", "cyberghost.com", "nokia.com", "co.uk",
                    "scholar.google.com", "isitdown.com", "tapinto.net",
                    "knownews.com", "makeuseof.com", "fordmuscle.com",
                    "marketbeat.com", "statetime.com"]
excluded_keywords = ["proxyDocument", ".pdf", "cryptojacking", "getting-started", "dig-in"]

for link in soup.find_all("a"):
    href = link.get("href")
    if href and href.startswith("/url"):
        parsed_href = urllib.parse.urlparse(href)
        query_params = urllib.parse.parse_qs(parsed_href.query)
        if "q" in query_params:
            final_link = query_params["q"][0]
            # 应用过滤规则
            if (final_link.startswith("https") and
                not any(domain in final_link for domain in excluded_domains) and
                not any(keyword in final_link for keyword in excluded_keywords)):
                print(final_link)

print(f"\nRequest URL: {url}")

内容的提问来源于stack exchange，提问作者Suryansh Agarwal