C#开发Google查重工具遇429错误：如何绕过或获取重试时间？

Hey there, let's work through this 429 error you're facing with your plagiarism checker. The issue boils down to Google's anti-scraping protections—your current requests look too "bot-like," and you're hitting rate limits without handling them properly. Let's break down the fixes step by step:

1. Fix Your Request Headers (Critical!) #

Your code tries to set a User-Agent but does it incorrectly. Google flags requests without a proper browser-like User-Agent immediately. Instead of adding it to Headers, set the UserAgent property directly, and add other common headers to mimic a real browser:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(SearchResults);
// Mimic a real Chrome browser on Windows
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Referer = "https://www.google.com/";

2. Extract Retry-After Time from 429 Responses #

When Google sends a 429 error, it usually includes a Retry-After header that tells you how many seconds to wait before retrying. You need to catch the specific WebException (not a generic Exception) to access this header:

catch (WebException ex)
{
    if (ex.Response is HttpWebResponse response && response.StatusCode == HttpStatusCode.TooManyRequests)
    {
        string retryAfter = response.Headers["Retry-After"];
        if (int.TryParse(retryAfter, out int secondsToWait))
        {
            MessageBox.Show($"Rate limited! Please wait {secondsToWait} seconds before trying again.", "Too Many Requests", MessageBoxButtons.OK, MessageBoxIcon.Warning);
            // You can automatically retry here using Thread.Sleep (sync) or Task.Delay (async)
            // Thread.Sleep(secondsToWait * 1000);
            // SearchAndWrite(text); // Retry the request
        }
        else
        {
            MessageBox.Show("Rate limited, but no retry time provided. Wait a few minutes and try again.", "Too Many Requests", MessageBoxButtons.OK, MessageBoxIcon.Warning);
        }
    }
    else
    {
        MessageBox.Show(ex.ToString(), "An error has occurred!", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
    }
}

3. Implement a Rate Limiting/Retry Strategy #

To avoid hitting 429 in the first place, add a delay between requests. If you do hit the limit, use an exponential backoff strategy (wait longer each time you get an error) instead of fixed delays:

private int retryAttempts = 0;
private void SearchAndWrite(string text)
{
    try
    {
        // ... your existing request and parsing code ...
        // Reset retry attempts on successful request
        retryAttempts = 0;
    }
    catch (WebException ex)
    {
        if (ex.Response is HttpWebResponse response && response.StatusCode == HttpStatusCode.TooManyRequests)
        {
            retryAttempts++;
            // Exponential backoff: wait 2^retryAttempts seconds
            int waitTime = (int)Math.Pow(2, retryAttempts);
            MessageBox.Show($"Rate limited! Retrying in {waitTime} seconds...", "Too Many Requests", MessageBoxButtons.OK, MessageBoxIcon.Warning);
            Thread.Sleep(waitTime * 1000);
            // Retry the request
            SearchAndWrite(text);
        }
        else
        {
            MessageBox.Show(ex.ToString(), "An error has occurred!", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
            retryAttempts = 0;
        }
    }
}

4. Switch to HttpClient (Modern Alternative) #

HttpWebRequest is outdated—HttpClient is better for managing connections, headers, and async operations. Here's a quick rewrite of your search logic using HttpClient:

private static readonly HttpClient _httpClient = new HttpClient();

// Initialize headers once in your form constructor
public YourFormName()
{
    InitializeComponent();
    // Set default headers to mimic a real browser
    _httpClient.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36");
    _httpClient.DefaultRequestHeaders.Accept.ParseAdd("text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    _httpClient.DefaultRequestHeaders.Referrer = new Uri("https://www.google.com/");
}

private async void SearchAndWrite(string text)
{
    listBox1.Items.Clear();
    // Properly escape the search query to handle special characters
    string searchUrl = $"http://google.com/search?q={Uri.EscapeDataString(text.Trim())}";
    try
    {
        HttpResponseMessage response = await _httpClient.GetAsync(searchUrl);
        response.EnsureSuccessStatusCode(); // Throws HttpRequestException on 4xx/5xx errors
        string htmlContent = await response.Content.ReadAsStringAsync();
        
        // Your HtmlAgilityPack parsing logic
        HtmlAgilityPack.HtmlDocument html = new HtmlAgilityPack.HtmlDocument();
        html.LoadHtml(htmlContent);
        // Handle null case if no links are found
        foreach (HtmlNode link in html.DocumentNode.SelectNodes("//a[@href]") ?? Enumerable.Empty<HtmlNode>())
        {
            string hrefValue = link.GetAttributeValue("href", string.Empty);
            if (!hrefValue.ToUpper().Contains("GOOGLE") && hrefValue.Contains("/url?q=") && hrefValue.ToUpper().Contains("HTTP://"))
            {
                int index = hrefValue.IndexOf("&");
                if (index > 0)
                {
                    hrefValue = hrefValue.Substring(0, index);
                    listBox1.Items.Add(hrefValue.Replace("/url?q=", ""));
                }
            }
        }
    }
    catch (HttpRequestException ex)
    {
        if (ex.StatusCode == HttpStatusCode.TooManyRequests)
        {
            double? secondsToWait = ex.Response.Headers.RetryAfter?.Delta?.TotalSeconds;
            if (secondsToWait.HasValue)
            {
                MessageBox.Show($"Rate limited! Wait {secondsToWait.Value:N0} seconds before retrying.", "Too Many Requests", MessageBoxButtons.OK, MessageBoxIcon.Warning);
                await Task.Delay(TimeSpan.FromSeconds(secondsToWait.Value));
                await SearchAndWrite(text);
            }
            else
            {
                MessageBox.Show("Rate limited. Wait a few minutes and try again.", "Too Many Requests", MessageBoxButtons.OK, MessageBoxIcon.Warning);
            }
        }
        else
        {
            MessageBox.Show(ex.ToString(), "An error has occurred!", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
        }
    }
}

Key Notes #

• Always use Uri.EscapeDataString() for your search query—your current code doesn't handle special characters (like spaces or ampersands) which can break the URL.
• Google's terms of service restrict scraping their search results. Make sure your tool complies with their policies to avoid getting your IP blocked long-term.

内容的提问来源于stack exchange，提问作者kahveciderin