如何用Python调用Google CSE API批量执行24000次多维度查询

Hey there! Since you already have the basic code to run a single query with Google's Custom Search API, we can expand that into a batch system with nested loops. Let's walk through this step by step, keeping it simple for Python beginners.

First, Organize Your Input Data #

First, we'll turn your 80 search phrases and 30 country domains into Python lists. Note: For the API, instead of targeting specific Google domains (like google.de), we can use the gl parameter (geolocation) to specify the region—this is the official, more reliable way to get country-specific results via the API. I'll map domains to their corresponding gl codes below.

# Your 80 search phrases (add all your phrases here)
search_phrases = [
    "PepsiCo scandal",
    "Samsung scandal",
    # ... add 78 more phrases
]

# Mapping country domains to geolocation codes (add your 30 here)
country_regions = {
    "google.de": "de",
    "google.co.uk": "uk",
    "google.com": "us",
    "google.fr": "fr",
    "google.jp": "jp",
    # ... add 25 more domain-code pairs
}

# Your API credentials (keep these secret!)
API_KEY = "your-google-api-key"
CX = "your-custom-search-engine-id"

Build the Nested Loop Logic #

We'll use three nested loops to cover all your requirements:

1. Loop through each search phrase
2. Loop through each country/region
3. Loop through each page (1 to 10)

For each page, we need to calculate the start parameter—Google CSE returns 10 results per page, so page 1 starts at 1, page 2 at 11, ..., page 10 at 91.

import requests
import time
import csv

# Set up a CSV file to save results (easy to analyze later)
with open("batch_search_results.csv", "w", newline="", encoding="utf-8") as csvfile:
    fieldnames = ["search_phrase", "country_domain", "page", "result_title", "result_link", "result_snippet"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()

    # Outer loop: iterate over each search phrase
    for phrase in search_phrases:
        print(f"Processing phrase: {phrase}")
        
        # Middle loop: iterate over each country/region
        for domain, gl_code in country_regions.items():
            print(f"  Targeting domain: {domain}")
            
            # Inner loop: iterate over each page (1 to 10)
            for page_num in range(1, 11):
                # Calculate the starting index for the current page
                start_index = (page_num - 1) * 10 + 1
                
                # Construct the API request URL
                url = f"https://www.googleapis.com/customsearch/v1?key={API_KEY}&cx={CX}&q={phrase}&gl={gl_code}&start={start_index}"
                
                try:
                    # Send the API request
                    response = requests.get(url)
                    response.raise_for_status()  # Trigger error for HTTP issues (like 404, 429)
                    results = response.json()
                    
                    # Extract and save each result to CSV
                    if "items" in results:
                        for item in results["items"]:
                            writer.writerow({
                                "search_phrase": phrase,
                                "country_domain": domain,
                                "page": page_num,
                                "result_title": item["title"],
                                "result_link": item["link"],
                                "result_snippet": item["snippet"]
                            })
                    else:
                        # Log when no results are found for the page
                        writer.writerow({
                            "search_phrase": phrase,
                            "country_domain": domain,
                            "page": page_num,
                            "result_title": "No results found",
                            "result_link": "",
                            "result_snippet": ""
                        })
                    
                    # Add a small delay to avoid hitting API rate limits
                    time.sleep(1)  # Adjust to 2 seconds if you get 429 errors
                    
                except Exception as e:
                    # Log errors instead of crashing the entire batch
                    print(f"    Error on page {page_num}: {str(e)}")
                    writer.writerow({
                        "search_phrase": phrase,
                        "country_domain": domain,
                        "page": page_num,
                        "result_title": f"Error: {str(e)}",
                        "result_link": "",
                        "result_snippet": ""
                    })
                    # Wait longer after an error to let the API recover
                    time.sleep(5)

print("Batch query completed! Results saved to batch_search_results.csv")

Key Notes for Your One-Time Batch #

• Quota Check: With 24,000 queries, confirm your paid API quota is sufficient. Google's paid CSE API costs $5 per 1,000 queries, so this batch will cost $120. Verify your quota in the Google Cloud Console before running.
• Rate Limits: Even with paid access, Google enforces rate limits (usually 100 queries per 100 seconds). The time.sleep(1) helps stay under this, but bump it to 2 seconds if you get 429 Too Many Requests errors.
• Credential Security: Keep your API_KEY and CX private—don't share this script publicly. For extra safety, you can load them from environment variables, but hardcoding is fine for a one-time run.
• Result Storage: I used CSV for simplicity, but you could switch to JSON or a database if you need more structured storage. CSV works great for opening in Excel/Google Sheets for analysis.

Quick Troubleshooting #

• If you get "invalid API key" errors, double-check your key and ensure the Custom Search API is enabled in your Google Cloud project.
• If results don't match country-specific Google sites, verify the gl code (e.g., au for Australia, in for India).
• Empty result pages are normal—they just mean there aren't 100 total results for that phrase in that region.

内容的提问来源于stack exchange，提问作者Chris_T