多搜索引擎笔记搜索Python代码故障：仅谷歌生效且结果异常

问题现象 #

• 仅Google执行搜索，但返回登录页、Cookie政策页等无效链接
• DuckDuckGo与Bing无法提取有效搜索结果链接

问题根源 #

1. 无请求头被拦截：直接发起requests.get请求未模拟浏览器，被搜索引擎识别为爬虫，返回非真实结果或拦截页面
2. 链接提取规则错误：
   • DuckDuckGo的结果链接class不是result__url，且部分是跳转链接需解析真实地址
   • Bing的结果链接未定位到正确的容器标签
   • Google直接提取所有http开头的a标签，包含大量无关导航链接，且未解析跳转链接
3. 查询字符串带换行符：readlines()读取的查询保留了换行符，导致搜索参数无效

修复后的代码 #

import requests
from bs4 import BeautifulSoup
import sqlite3
from urllib.parse import unquote, urlparse

# 模拟浏览器请求头，避免被拦截
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

# 读取查询语句，去除换行符
with open('search_queries.txt', 'r', encoding='utf-8') as file:
    queries = [q.strip() for q in file.readlines() if q.strip()]

# 连接SQLite数据库
conn = sqlite3.connect('unique_links.db')
cursor = conn.cursor()
cursor.execute('''CREATE TABLE IF NOT EXISTS links (url TEXT UNIQUE)''')
conn.commit()

def extract_real_url(link, engine):
    """解析搜索引擎的跳转链接，获取真实地址"""
    if engine == 'google' and link.startswith('/url?q='):
        return unquote(link.split('&')[0].replace('/url?q=', ''))
    elif engine == 'duckduckgo' and link.startswith('/l/?uddg='):
        return unquote(link.split('&')[0].replace('/l/?uddg=', ''))
    else:
        # 检查是否为有效http链接
        parsed = urlparse(link)
        return link if parsed.scheme in ['http', 'https'] else None

def search_and_collect_links(query, search_engine):
    try:
        # 构造搜索URL
        if search_engine == 'duckduckgo':
            search_url = f'https://duckduckgo.com/html/?q={query}'
        elif search_engine == 'google':
            search_url = f'https://www.google.com/search?q={query}'
        elif search_engine == 'bing':
            search_url = f'https://www.bing.com/search?q={query}'
        else:
            return

        # 发起请求
        response = requests.get(search_url, headers=HEADERS, timeout=10)
        response.raise_for_status()  # 抛出HTTP错误
        soup = BeautifulSoup(response.text, 'html.parser')

        links = []
        # 针对不同引擎提取结果链接
        if search_engine == 'google':
            # 定位到搜索结果容器，提取真实链接
            result_containers = soup.find_all('div', class_='g')
            for container in result_containers:
                a_tag = container.find('a', href=True)
                if a_tag:
                    real_url = extract_real_url(a_tag['href'], 'google')
                    if real_url:
                        links.append(real_url)
        elif search_engine == 'duckduckgo':
            # 定位DuckDuckGo的结果链接
            result_links = soup.find_all('a', class_='result__a', href=True)
            for a_tag in result_links:
                real_url = extract_real_url(a_tag['href'], 'duckduckgo')
                if real_url:
                    links.append(real_url)
        elif search_engine == 'bing':
            # 定位Bing的结果容器
            result_containers = soup.find_all('li', class_='b_algo')
            for container in result_containers:
                a_tag = container.find('a', href=True)
                if a_tag:
                    links.append(a_tag['href'])

        # 批量插入数据库，提升效率
        if links:
            cursor.executemany('INSERT OR IGNORE INTO links (url) VALUES (?)', [(link,) for link in links])
            conn.commit()
        print(f"{search_engine} 搜索 '{query}' 成功，收集到 {len(links)} 个有效链接")
    except Exception as e:
        print(f"{search_engine} 搜索 '{query}' 失败: {str(e)}")

# 遍历所有查询和搜索引擎
for query in queries:
    for engine in ['duckduckgo', 'google', 'bing']:
        search_and_collect_links(query, engine)

# 导出到txt文件
with open('unique_links.txt', 'w', encoding='utf-8') as link_file:
    cursor.execute('SELECT url FROM links')
    for link in cursor.fetchall():
        link_file.write(link[0] + '\n')

# 关闭数据库连接
conn.close()
print("所有任务完成，结果已保存到unique_links.db和unique_links.txt")

关键修复说明 #

• 添加请求头：模拟浏览器标识，避免被搜索引擎拦截
• 修正链接提取规则：针对每个搜索引擎的页面结构，定位到真实结果容器解析链接
• 解析跳转链接：处理Google和DuckDuckGo的跳转链接，提取真实目标地址
• 处理查询字符串：去除换行符，确保搜索参数有效
• 批量数据库操作：减少提交次数，提升效率
• 异常处理：捕获请求错误，避免单个搜索失败导致程序终止

内容的提问来源于stack exchange，提问作者Kavis