C# HttpWebRequest美国IP超时问题：浏览器正常但代码请求失败

Hey there, I’ve dealt with similar headaches with fashion e-commerce sites like these—their anti-bot and geo-based security rules can be surprisingly finicky, especially when switching IP regions. Let’s break down the most likely fixes to get your requests working from US IPs:

1. Fix Low-Level Connection Settings (HttpWebRequest’s Defaults Are Too "Robotic") #

HttpWebRequest has some outdated default behaviors that trigger red flags on strict sites. Tweak these first:

• Disable Expect100Continue (most modern sites don’t use this, and it adds an extra round-trip that can be flagged):

  ServicePointManager.Expect100Continue = false;
  

• Force modern TLS versions (old protocols will get blocked immediately):

  ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;
  

• Adjust timeouts to account for potential US-based latency, and enable keep-alive to mimic browser behavior:

  request.Timeout = 30000; // 30 seconds
  request.ReadWriteTimeout = 30000;
  request.KeepAlive = true;
  

2. Fill in All "Modern Browser" Request Headers #

You mentioned trying header combinations, but many sites now check for newer, less obvious headers that browsers send. Add these to your request:

request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers["Accept-Encoding"] = "gzip, deflate, br";
request.Headers["Accept-Language"] = "en-US,en;q=0.9";
request.Referer = "https://www.mrporter.com/"; // Match the site you're targeting
request.Headers["DNT"] = "1";
request.Headers["Upgrade-Insecure-Requests"] = "1";
// Add these Sec-Fetch headers—they're a big red flag if missing
request.Headers["Sec-Fetch-Dest"] = "document";
request.Headers["Sec-Fetch-Mode"] = "navigate";
request.Headers["Sec-Fetch-Site"] = "none";

3. Audit Your US Proxy IP Quality #

This is the #1 culprit for US IP failures:

• Shared proxies are almost certainly already blocked by these sites. Switch to a residential proxy (not datacenter) that’s dedicated to your use.
• Test the proxy with a simple curl command first to rule out connectivity issues:

  curl -x http://your-us-proxy:port https://www.mrporter.com/ -v
  

  If curl fails too, the proxy is either blocked or has network issues.

4. Mimic Human Request Rhythm #

US-based anti-bot systems often enforce stricter rate limits. Add random delays between requests (1-3 seconds) and avoid spamming requests in quick succession. For example:

// Add this before your next request
Thread.Sleep(new Random().Next(1000, 3000));

5. Switch to HttpClient (It’s More Browser-Friendly) #

HttpWebRequest is legacy technology—HttpClient’s underlying implementation handles connection pooling and modern web standards better. Here’s a quick example:

var handler = new HttpClientHandler
{
    AllowAutoRedirect = true,
    UseCookies = true, // Automatically manages session cookies
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli
};

using var client = new HttpClient(handler);
// Set default headers once
client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/118.0.0.0 Safari/537.36");
client.DefaultRequestHeaders.Accept.ParseAdd("text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
client.DefaultRequestHeaders.AcceptEncoding.ParseAdd("gzip, deflate, br");
client.DefaultRequestHeaders.AcceptLanguage.ParseAdd("en-US,en;q=0.9");
client.DefaultRequestHeaders.Referrer = new Uri("https://www.mrporter.com/");
client.Timeout = TimeSpan.FromSeconds(30);

// Make your request
var response = await client.GetAsync("https://www.mrporter.com/");
response.EnsureSuccessStatusCode();
var content = await response.Content.ReadAsStringAsync();

6. Verify Cookie Handling #

Some sites require session cookies set on the first request to allow subsequent access. With HttpWebRequest, you need to explicitly use a CookieContainer:

var cookieJar = new CookieContainer();
request.CookieContainer = cookieJar;
// Reuse this cookieJar for follow-up requests to the same site

If none of these work, use a tool like Fiddler to capture a successful request from a real browser on the same US IP, then compare every header and connection detail to your code’s request. The difference is almost always something small you missed.

内容的提问来源于stack exchange，提问作者Turok