Python urllib请求出现HTTP Error 404: Not Found问题求助

Hey there, let's work through this 404 error you're hitting when using urllib. A 404 means the server can't locate the resource you're requesting, so the root issue almost always ties back to your URL or how you're sending the request. Here's how to troubleshoot and fix it step by step:

1. First, Verify Your URL is 100% Correct #

This is the most common culprit—start here:

• Copy the exact URL you're using in your code into a browser and try accessing it. If you get a 404 here too, the URL itself is invalid. Look for typos, missing path segments, or outdated domain names (maybe the website moved its resources).
• Check for special characters in the URL (like spaces, Chinese characters, or symbols). urllib doesn't automatically encode these, which can break the request. Use urllib.parse.quote() to properly encode the URL while keeping critical parts like the protocol (http/https) intact:

  import urllib.request
  import urllib.parse
  
  original_url = "https://example.com/天气 forecast"
  encoded_url = urllib.parse.quote(original_url, safe=':/')
  data = urllib.request.urlopen(encoded_url).read()
  

• Confirm the protocol (http vs https). Some websites only support one—if you use http for a site that requires https, you'll get a 404 (or a redirect error).

2. Add Proper Request Headers #

Many websites block requests that don't include a valid User-Agent header (they flag it as a bot). Try mimicking a browser's request to get past this:

import urllib.request

url = "your_valid_url_here"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

# Create a request object with the headers
req = urllib.request.Request(url, headers=headers)
data = urllib.request.urlopen(req).read()

3. Check if You're Using the Right Request Method #

If the resource expects a POST request (like form submissions or API endpoints) but you're sending a GET (urllib's default), you'll get a 404. To send a POST request:

import urllib.request
import urllib.parse

url = "your_post_endpoint_here"
# Encode your POST data
post_data = urllib.parse.urlencode({'key': 'value'}).encode('utf-8')
req = urllib.request.Request(url, data=post_data, method='POST')
data = urllib.request.urlopen(req).read()

4. Rule Out Dynamic URL Issues #

If your URL includes dynamic elements (like session IDs, timestamps, or API keys), those might have expired or be formatted incorrectly. Double-check that all parameters are up-to-date and match what the server expects.

Start with the first step—testing the URL in your browser. That's the quickest way to confirm if the problem is with the URL itself or your request setup.

内容的提问来源于stack exchange，提问作者Tolly Boy101