如何用Java（含Android）向Google Chrome图片发送请求并获取结果HTML？

Hey there! Let's tackle your problem: building a tool that takes a keyword, fetches the full HTML of Google Image Search results (matching what you'd see in Chrome), plus the Android-specific Java version. Here's how to do it:

Core Concepts First #

To get the same HTML as a browser, you need to mimic real browser behavior—Google has anti-scraping measures that block plain HTTP requests. Key steps:

• Use the correct Google Image Search URL: https://www.google.com/search?q={YOUR_KEYWORD}&tbm=isch (the tbm=isch flag tells Google to return image results).
• Send browser-like request headers: Especially the User-Agent (to pretend you're using Chrome/Edge), plus Accept and Accept-Language headers.
• Handle cookies if needed: Sometimes Google requires a valid session cookie to serve results without a CAPTCHA.

Android (Java) Implementation #

Below are two solid options for Android—one using a popular library (OkHttp) for simplicity, and another using Android's native tools if you want to avoid dependencies.

Option 1: Use OkHttp (Recommended) #

OkHttp makes HTTP requests way cleaner. First, add the dependency to your app-level build.gradle:

implementation 'com.squareup.okhttp3:okhttp:4.11.0'

Then the core code:

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import java.io.IOException;

public class GoogleImageScraper {
    private static final String BASE_SEARCH_URL = "https://www.google.com/search?tbm=isch&q=";
    // Mimic a desktop Chrome user-agent to avoid being blocked
    private static final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36";

    public static String fetchSearchResultsHtml(String keyword) throws IOException {
        // Encode the keyword to handle spaces/special characters
        String encodedKeyword = java.net.URLEncoder.encode(keyword, "UTF-8");
        String fullUrl = BASE_SEARCH_URL + encodedKeyword;

        OkHttpClient client = new OkHttpClient();
        Request request = new Request.Builder()
                .url(fullUrl)
                .header("User-Agent", USER_AGENT)
                .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
                .header("Accept-Language", "en-US,en;q=0.5")
                .build();

        try (Response response = client.newCall(request).execute()) {
            if (!response.isSuccessful()) {
                throw new IOException("Request failed with code: " + response.code());
            }
            return response.body().string();
        }
    }
}

Option 2: Native HttpURLConnection (No Dependencies) #

If you prefer to stick with Android's built-in tools, use HttpURLConnection:

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;

public class NativeGoogleImageScraper {
    private static final String BASE_SEARCH_URL = "https://www.google.com/search?tbm=isch&q=";
    private static final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36";

    public static String fetchSearchResultsHtml(String keyword) throws Exception {
        String encodedKeyword = URLEncoder.encode(keyword, "UTF-8");
        URL url = new URL(BASE_SEARCH_URL + encodedKeyword);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();

        // Set browser-like headers
        connection.setRequestMethod("GET");
        connection.setRequestProperty("User-Agent", USER_AGENT);
        connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        connection.setRequestProperty("Accept-Language", "en-US,en;q=0.5");

        // Read the response HTML
        BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        StringBuilder htmlContent = new StringBuilder();
        String line;
        while ((line = reader.readLine()) != null) {
            htmlContent.append(line);
        }
        reader.close();
        connection.disconnect();

        return htmlContent.toString();
    }
}

Critical Notes #

1. Anti-Scraping Heads-Up: Google changes its anti-scraping rules often. If you get a 403 error or a CAPTCHA page, you might need to:
   • Add cookie persistence (store cookies from an initial Google homepage request).
   • Use rotating proxy IPs.
   • Consider Google's official Custom Search API (requires an API key, has rate limits, but is way more reliable than scraping).
2. Android Permissions: Don't forget to add internet access to your AndroidManifest.xml:

   <uses-permission android:name="android.permission.INTERNET" />
   

3. Threading Rule: Android won't let you run network requests on the main thread. Wrap the above code in a Coroutine, AsyncTask, or ExecutorService to avoid crashes.

内容的提问来源于stack exchange，提问作者maksonnie