使用urllib请求谷歌搜索时遇HTTP Error 405，URL参数解析失效求助

Hey Jay, I’ve run into this exact snag before—Google’s search engine blocks requests that don’t include a proper user agent header, since it’s trying to filter out automated scraping attempts. That’s almost certainly why you’re hitting that HTTP error.

Here’s how to tweak your code to get past this:

1. Add a User-Agent header to mimic a real browser request. This tells Google your request is coming from a legitimate user, not a script.
2. Wrap your request in error handling to get clearer feedback if something still goes wrong.

Modified Working Code #

import urllib.parse
import urllib.request

url = 'https://www.google.com/search'
values = {'q': 'python programming tutorials'}
data = urllib.parse.urlencode(values)
data = data.encode('utf-8')

# Mimic a browser request with a valid User-Agent
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

request = urllib.request.Request(url, data, headers=headers)
try:
    response = urllib.request.urlopen(request)
    # Read and print the response content
    response_content = response.read().decode('utf-8')
    print(response_content)
except urllib.error.HTTPError as e:
    print(f"HTTP Error: {e.code} - {e.reason}")
except urllib.error.URLError as e:
    print(f"URL Error: {e.reason}")

Quick Notes: #

• The User-Agent string above is for a Chrome browser on macOS—you can grab your own by searching "what's my user agent" in your browser if you want to match your system exactly.
• The try/except blocks will help you pinpoint specific issues (like 403 Forbidden if the user agent is rejected, or no internet connection) instead of getting a vague error.
• Heads up: Google’s anti-scraping rules are strict. Even with a user agent, you might hit rate limits if you make too many requests fast. For regular use, consider Google’s official Search API to avoid these headaches long-term.

内容的提问来源于stack exchange，提问作者Jay