使用Python Requests进行网页爬取返回乱码问题求助

Hey there! As someone who’s worked through tons of beginner scraping hurdles, let’s figure out why your code isn’t returning the HTML you expect—and fix it fast.

The Root Issue #

Most modern websites have basic anti-scraping checks, and the requests library sends a default User-Agent header that shouts "I’m a bot!" (it looks like python-requests/[version number]). That storage site is probably blocking your request and sending back garbled/truncated HTML as a result.

Quick Fix: Mimic a Real Browser’s Request #

The easiest solution is to add request headers that match what a typical web browser sends. Here’s how to update your code:

import requests

# Headers copied from a standard Chrome browser request
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
}

target_url = "https://www.usstoragecenters.com/storage-units/fl/north-miami-beach/15555-w-dixie-hwy"
page = requests.get(target_url, headers=headers)

# First, confirm the request succeeded (200 = OK)
print(f"Response Status: {page.status_code}")

# Now check the proper HTML content
print(page.text)

Key Tips to Keep in Mind: #

• Always check the status code first: If you see 403 Forbidden, that’s proof the site blocked your bot-like request. A 200 OK means your modified request worked.
• User-Agent is non-negotiable: This header tells the server what kind of client is accessing it. Using a browser’s User-Agent bypasses 90% of basic anti-scraping filters.
• If HTML still looks off: Some sites load prices (and other content) dynamically with JavaScript. If that’s the case here, you’ll need tools like Selenium or Playwright to simulate a full browser that runs JS. But start with the header fix—it’s the simplest first step.

内容的提问来源于stack exchange，提问作者t25