求助：使用JSoup解析谷歌天气页面时.data()方法的正确调用

Hey there! Let’s work through your JSoup issue with scraping Google's weather results—specifically figuring out how to use the .data() method correctly. I’ve been in your shoes before, so I know how frustrating it can be when you’ve got the framework set up but can’t nail down the element targeting.

First, let’s clarify what .data() does, since that’s your main pain point:

• If you call element.data() without any arguments, it returns a Map of all the element’s data-* attributes (e.g., an element with data-temp="22" and data-condition="sunny" would give you a map with keys "temp" and "condition").
• When you pass a key like element.data("temp"), it directly returns the value of the data-temp attribute—this is the method you’ll want for extracting specific weather details.

Step 1: Get the Page (With a Critical Fix) #

Google blocks requests that don’t look like they’re coming from a browser, so you must set a User-Agent in your JSoup connection. Here’s how to do that:

Document doc = Jsoup.connect("https://www.google.com/search?q=weather")
                .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
                .timeout(10000)
                .get();

Step 2: Target the Weather Container #

Google’s HTML structure changes often, but as of 2024, the main weather card lives in a div with the class wob_dcp. Use that to narrow down your search:

Element weatherCard = doc.selectFirst("div.wob_dcp");
if (weatherCard == null) {
    System.out.println("Oops—Google probably updated their HTML structure. Check the element classes with your browser's dev tools!");
    return;
}

Step 3: Use .data() to Extract Weather Details #

Now you can dig into the specific elements inside the weather card. Each key piece of info is stored in a data-* attribute. For example:

• Current temperature is in data-wob-tm
• Weather condition (like "Sunny") is in data-wob-desc
• Humidity is in data-wob-hm

Here’s how to pull those values with .data():

// Get current temperature
String currentTemp = weatherCard.selectFirst("span[data-wob-tm]").data("wob-tm");
// Get weather description
String weatherCondition = weatherCard.selectFirst("span[data-wob-desc]").data("wob-desc");
// Get humidity percentage
String humidity = weatherCard.selectFirst("span[data-wob-hm]").data("wob-hm");

Full Working Example #

Putting it all together, your code might look like this:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public class GoogleWeatherScraper {
    public static void main(String[] args) {
        try {
            // Fetch the Google weather page
            Document doc = Jsoup.connect("https://www.google.com/search?q=weather")
                    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
                    .timeout(10000)
                    .get();

            // Locate the main weather card
            Element weatherCard = doc.selectFirst("div.wob_dcp");
            if (weatherCard == null) {
                System.out.println("Couldn't find the weather card—Google's HTML might have changed.");
                return;
            }

            // Extract details using .data()
            String temp = weatherCard.selectFirst("span[data-wob-tm]").data("wob-tm");
            String condition = weatherCard.selectFirst("span[data-wob-desc]").data("wob-desc");
            String humidity = weatherCard.selectFirst("span[data-wob-hm]").data("wob-hm");

            // Print results
            System.out.println("Current Weather:");
            System.out.println("Temperature: " + temp + "°C");
            System.out.println("Condition: " + condition);
            System.out.println("Humidity: " + humidity);

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Important Notes #

1. Google’s HTML changes frequently: If this code stops working, right-click the weather element in your browser, select "Inspect", and check the updated class names and data-* attributes.
2. Always use a valid User-Agent: Without it, Google will return an error page instead of the search results.
3. Handle null elements: Always check if selectFirst() returns null to avoid NullPointerExceptions when the structure changes.

内容的提问来源于stack exchange，提问作者CalebDunham