如何用BeautifulSoup抓取谷歌天气特定文本？排查空结果问题

想要用Python的BeautifulSoup获取谷歌天气页面中类似“New York City, New York, USA”的城市全称文本，但当前代码失效。尝试使用soup.find_all('div', attrs={'class': 'wob_loc q8U8x'})或soup.find_all('div', attrs={'id': 'wob_loc'})均返回空列表（.find方法返回None），怀疑是get_html_content函数异常或被谷歌拦截。

当前代码如下：

from django.shortcuts import render
import requests

def get_html_content(city):
    USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36"
    LANGUAGE = "en-US,en;q=0.5"
    session = requests.Session()
    session.headers['User-Agent'] = USER_AGENT
    session.headers['Accept-Language'] = LANGUAGE
    session.headers['Content-Language'] = LANGUAGE
    city.replace(" ", "+")
    html_content = session.get(f"https://www.google.com/search?q=weather+in+{city}").text
    return html_content

def home(request):
    result = None
    if 'city' in request.GET: 
        city = request.GET.get('city')
        html_content = get_html_content(city)
        from bs4 import BeautifulSoup
        soup = BeautifulSoup(html_content, 'html.parser')
        soup.find_all('div', attrs={'class': 'wob_loc q8U8x'})
        # OR
        soup.find_all('div', attrs={'id': 'wob_loc'})

• 修复城市名称替换逻辑：Python字符串是不可变类型，city.replace(" ", "+")不会修改原变量，必须将结果赋值回变量：city = city.replace(" ", "+")
• 更新请求头中的User-Agent：旧版Chrome的UA容易被谷歌拦截，换成较新的浏览器UA，比如：Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
• 添加搜索参数锁定界面语言：在请求URL中加入hl=en参数，强制返回英文界面，确保目标元素的class/id与你查找的规则匹配
• 验证返回的HTML内容：先打印html_content的前几百字符，确认是否拿到了正常的搜索结果页面，而非谷歌的人机验证或拦截页面
• 更换BeautifulSoup解析器：将html.parser换成lxml（需提前安装：pip install lxml），对复杂HTML的解析更稳定

from django.shortcuts import render
import requests
from bs4 import BeautifulSoup  # 建议将导入放在文件顶部，避免重复导入

def get_html_content(city):
    # 更新为较新的User-Agent
    USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
    LANGUAGE = "en-US,en;q=0.5"
    session = requests.Session()
    session.headers['User-Agent'] = USER_AGENT
    session.headers['Accept-Language'] = LANGUAGE
    session.headers['Content-Language'] = LANGUAGE
    # 修复字符串替换逻辑
    city = city.replace(" ", "+")
    # 添加hl=en参数，强制英文界面
    url = f"https://www.google.com/search?q=weather+in+{city}&hl=en"
    response = session.get(url)
    # 检查请求是否成功，失败则抛出异常
    response.raise_for_status()
    return response.text

def home(request):
    result = None
    if 'city' in request.GET: 
        city = request.GET.get('city')
        html_content = get_html_content(city)
        # 使用lxml解析器
        soup = BeautifulSoup(html_content, 'lxml')
        # 通过id定位目标元素
        location_element = soup.find('div', id='wob_loc')
        if location_element:
            result = location_element.text.strip()
            # 此处可将result传递给模板或进行后续处理
    return render(request, 'your_template.html', {'result': result})

内容的提问来源于stack exchange，提问作者Peter