使用Google News RSS链接爬取文章时，如何处理Google授权同意页面？

你遇到的问题是Google基于隐私政策要求，会将未携带有效同意Cookie的请求重定向到Cookie授权页，导致直接用requests无法获取目标文章内容。以下是几种可行的处理方案：

方案1：携带已同意的Cookie访问 #

手动完成一次Cookie同意操作后，提取浏览器中的Cookie并在请求中携带，这是最简单的临时解决方案：

1. 用浏览器打开目标链接，点击"同意"完成Cookie授权；
2. 打开浏览器开发者工具（F12）→ Network标签，刷新页面后找到主请求，复制Cookie请求头的值；
3. 将Cookie添加到requests的headers中。

修改后的代码示例：

import requests
from bs4 import BeautifulSoup

google_rss_url = 'https://news.google.com/rss/articles/CBMimwFBVV95cUxNVmJMNUdiamVCNkJSb1E4NVU0SlBFQUNneXpEaHFuRUJpN3lwRXFNNGdRalpITmFUQUh4Z3lsOVZ4ekFSdWVwVEljVUJOT241S1g2dmRmd3NnRmJjamU4TVFFdUVXd0N2MGVPTUdxb0RVZ2xQbUlkS1Y3eEhKbmdBN2hSUHNzS2ZucjlKQl84SW13ZVpXYlZXRnRSZw?oc=5'

headers = {
    'User-Agent': 'Mozilla/5.0',
    # 替换为你从浏览器复制的Cookie字符串
    'Cookie': 'YOUR_COOKIE_STRING_HERE'
}

response = requests.get(google_rss_url, headers=headers)

if 'Cookie Notice' not in response.text:
    soup = BeautifulSoup(response.content, 'html.parser')
    paragraphs = soup.find_all('p')
    article_text = '\n'.join([para.get_text() for para in paragraphs])
    print(article_text)
else:
    print("Cookie已失效，请重新获取")

注意：Cookie会定期过期，需要重新手动获取。

方案2：程序化提交Cookie同意表单 #

通过解析授权页面的表单参数，自动提交同意请求，无需手动操作：

import requests
from bs4 import BeautifulSoup

google_rss_url = 'https://news.google.com/rss/articles/CBMimwFBVV95cUxNVmJMNUdiamVCNkJSb1E4NVU0SlBFQUNneXpEaHFuRUJpN3lwRXFNNGdRalpITmFUQUh4Z3lsOVZ4ekFSdWVwVEljVUJOT241S1g2dmRmd3NnRmJjamU4TVFFdUVXd0N2MGVPTUdxb0RVZ2xQbUlkS1Y3eEhKbmdBN2hSUHNzS2ZucjlKQl84SW13ZVpXYlZXRnRSZw?oc=5'

headers = {'User-Agent': 'Mozilla/5.0'}
session = requests.Session()

# 1. 获取授权页面
response = session.get(google_rss_url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')

# 检查是否需要授权
form = soup.find('form', {'action': 'https://consent.google.com/s'})
if not form:
    # 已跳过授权，直接解析文章
    paragraphs = soup.find_all('p')
    article_text = '\n'.join([para.get_text() for para in paragraphs])
    print(article_text)
else:
    # 2. 提取表单参数
    form_data = {}
    for input_tag in form.find_all('input'):
        if input_tag.get('name') and input_tag.get('value'):
            form_data[input_tag['name']] = input_tag['value']
    # 设置同意选项
    form_data['set_eom'] = 'true'
    form_data['set_aps'] = 'true'

    # 3. 提交同意表单
    consent_response = session.post('https://consent.google.com/s', data=form_data, headers=headers)

    # 4. 再次请求原链接，此时已携带同意Cookie
    article_response = session.get(google_rss_url, headers=headers)
    article_soup = BeautifulSoup(article_response.content, 'html.parser')
    paragraphs = article_soup.find_all('p')
    article_text = '\n'.join([para.get_text() for para in paragraphs])
    print(article_text)

注意：Google可能会调整表单结构，导致此方案失效，需定期检查表单参数。

方案3：使用Headless浏览器（最稳定） #

用Selenium模拟真实浏览器行为，自动点击同意按钮，能有效应对Google的反爬机制：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

google_rss_url = 'https://news.google.com/rss/articles/CBMimwFBVV95cUxNVmJMNUdiamVCNkJSb1E4NVU0SlBFQUNneXpEaHFuRUJpN3lwRXFNNGdRalpITmFUQUh4Z3lsOVZ4ekFSdWVwVEljVUJOT241S1g2dmRmd3NnRmJjamU4TVFFdUVXd0N2MGVPTUdxb0RVZ2xQbUlkS1Y3eEhKbmdBN2hSUHNzS2ZucjlKQl84SW13ZVpXYlZXRnRSZw?oc=5'

# 配置无头Chrome
chrome_options = Options()
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--user-agent=Mozilla/5.0')

driver = webdriver.Chrome(options=chrome_options)
driver.get(google_rss_url)

try:
    # 等待并点击同意按钮
    accept_button = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), '同意') or contains(text(), 'Accept')]"))
    )
    accept_button.click()

    # 等待文章加载完成
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, 'p'))
    )

    # 解析文章内容
    page_source = driver.page_source
    soup = BeautifulSoup(page_source, 'html.parser')
    paragraphs = soup.find_all('p')
    article_text = '\n'.join([para.get_text() for para in paragraphs])
    print(article_text)
finally:
    driver.quit()

注意：需要安装selenium库和对应浏览器的驱动（如ChromeDriver），且频繁爬取需添加请求间隔，避免IP被封禁。

内容的提问来源于stack exchange，提问作者Jurgita-ds