如何通过Requests获取Google Trends搜索结果CSV文件的下载URL

Great question! Ditching Selenium for Requests is a solid call for speed and reliability—here's a step-by-step breakdown to capture that download URL and automate the process smoothly:

1. Capture the Download Request via Browser Dev Tools #

First, you need to catch the exact request your browser sends when clicking the CSV button:

• Load up Google Trends, run your search, and navigate to the page with the download button.
• Open your browser's DevTools (hit F12 or Ctrl+Shift+I), then switch to the Network tab.
• Check the "Preserve log" option at the top—this prevents the request list from clearing when the download starts.
• Click the CSV download button. Look for a new GET request in the Network list that ends with /csv (like the example URL you shared).
• Right-click that request, then choose "Copy > Copy as cURL (bash)" or just copy the full URL directly. This gives you the exact endpoint and parameters you need.

2. Understand the URL's Key Components #

That URL has two non-negotiable parts you need to handle:

• req parameter: This is a URL-encoded JSON object containing all your search criteria—time range, location, keywords, category, etc. You can decode it using urllib.parse.unquote() to see the plain JSON, then modify or reconstruct it for your own requests.
• token parameter: This is a temporary validation token generated by Google. It expires quickly, so you can't hardcode it—you'll need to extract it from the Google Trends page each time you run your script.

3. Automate the Process with Requests #

Here's a working example script to fetch the token, construct the CSV URL, and download the file:

import requests
import json
import urllib.parse
import re

# Set up your search parameters
search_query = "bahria town karachi"
geo_location = "PK"
time_range = "2021-03-09 2022-03-09"
category = 29  # Adjust this to your category ID (29 is Real Estate)

# Mimic a browser's headers to avoid being blocked
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Referer": "https://trends.google.com/"
}

# Step 1: Fetch the initial Trends page to extract the token
explore_url = f"https://trends.google.com/trends/explore?q={urllib.parse.quote(search_query)}&geo={geo_location}&date={urllib.parse.quote(time_range)}"
response = requests.get(explore_url, headers=headers)

# Extract the token using regex (it's embedded in the page's script content)
token_match = re.search(r'token: "([^"]+)"', response.text)
if not token_match:
    raise ValueError("Could not find the authentication token—page structure might have changed")
token = token_match.group(1)

# Step 2: Construct the encoded 'req' parameter
req_payload = {
    "time": time_range,
    "resolution": "WEEK",
    "locale": "en-US",
    "comparisonItem": [
        {
            "geo": {"country": geo_location},
            "complexKeywordsRestriction": {
                "keyword": [{"type": "BROAD", "value": search_query}]
            }
        }
    ],
    "requestOptions": {"property": "", "backend": "IZG", "category": category}
}
encoded_req = urllib.parse.quote(json.dumps(req_payload))

# Step 3: Build the CSV download URL and fetch the file
csv_url = f"https://trends.google.com/trends/api/widgetdata/multiline/csv?req={encoded_req}&token={token}&tz=-300"
csv_response = requests.get(csv_url, headers=headers)

# Save the CSV to disk
with open("google_trends_data.csv", "w", encoding="utf-8") as file:
    file.write(csv_response.text)

print("CSV downloaded successfully!")

4. Critical Tips to Avoid Issues #

• Keep headers updated: Google blocks requests that don't look like they're coming from a real browser. Always use a valid User-Agent and include the Referer header.
• Token expires quickly: Don't try to reuse a token from a previous session—extract it fresh every time you run the script.
• Adjust the widget path: The /multiline/ part of the URL corresponds to the type of data you're downloading (e.g., geoMap for regional data). If your CSV download uses a different path, update it accordingly.
• Handle 403 errors: If you get a 403 Forbidden response, check your headers against the ones your browser sends (you can copy them from DevTools) or add a small delay between requests to avoid triggering rate limits.

内容的提问来源于stack exchange，提问作者Abdul Moiz.