使用BeautifulSoup爬取天气网站无法获取服务端温度数值求助

Hey there! I see you're having trouble grabbing the actual temperature for Dhaka from The Weather Network using BeautifulSoup and requests. Let's break down why this is happening and how to fix it.

Why Your Current Code Isn't Working #

The temperature value on this site is dynamically loaded with JavaScript. When you use requests.get(), you're only fetching the static initial HTML of the page—this doesn't include the real temperature data, which gets pulled in by the site's JavaScript after the page loads. That's why you're only getting the empty <span>°</span> tag instead of the actual number.

Solution 1: Use Selenium to Simulate a Browser #

Selenium lets you launch a real browser, wait for the JavaScript to render the content, then extract the temperature. Here's how to do it:

First, install Selenium and make sure you have the right browser driver (like ChromeDriver for Chrome):

pip install selenium

Then use this code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Initialize Chrome browser (replace with your preferred browser driver)
driver = webdriver.Chrome()
driver.get("https://www.theweathernetwork.com/bd/weather/dhaka/dhaka")

try:
    # Wait up to 10 seconds for the temperature element to load
    temp_element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".current-obs .temp"))
    )
    # Get the full temperature text (e.g., "28°")
    current_temp = temp_element.text
    print(f"Current temperature in Dhaka: {current_temp}")
finally:
    # Make sure to close the browser when done
    driver.quit()

This code waits for the JavaScript to finish loading the temperature before extracting it, so you'll get the real value instead of just the degree symbol.

Solution 2: Directly Call the Site's API (More Efficient) #

A faster, lighter alternative is to find the API endpoint the site uses to fetch weather data. Here's how to do it:

1. Open the weather page in your browser and hit F12 to open DevTools.
2. Go to the Network tab, filter by "XHR" or "Fetch".
3. Refresh the page and look for requests that return weather data (look for keywords like "current" or "temperature").
4. Copy the API URL and use requests to fetch the JSON data directly.

Example code (you'll need to replace the API URL with the real one you find):

import requests

# Replace with the actual API endpoint you find in DevTools
api_url = "https://api.theweathernetwork.com/api/v1/weather/current?location=dhaka"
response = requests.get(api_url)
weather_data = response.json()

# Extract temperature from the JSON (adjust the key based on the actual response structure)
current_temp = weather_data["data"]["current"]["temperature"]
print(f"Current temperature in Dhaka: {current_temp}°")

This method avoids loading a full browser, so it's faster and uses fewer resources. Just note that API endpoints can change over time, so you might need to recheck DevTools if it stops working.

内容的提问来源于stack exchange，提问作者HMBR IT