C#中HttpWebRequest内存占用过高问题咨询

Hey there, let's dive into your memory issue first—it's definitely the bigger priority here—then we'll touch on that switch statement question.

Addressing High Memory Usage from Response Streams #

Your hunch is right: the line string _responseData = new StreamReader(response.GetResponseStream()).ReadToEnd(); is almost certainly the culprit. Reading the entire response body into a single string shoves all that data into memory at once, which adds up fast when you're running hundreds/thousands of requests. Plus, your current code isn't properly disposing of resources, which leads to memory leaks over time. Here's how to fix it:

1. Use using Statements to Auto-Dispose Resources #

Both HttpWebRequest and HttpWebResponse implement IDisposable, so wrapping them in using blocks ensures they release connections and memory as soon as you're done with them. Same goes for the StreamReader—don't leave it hanging around.

2. Avoid Reading the Entire Response at Once #

Instead of loading the full response into a single string, process it incrementally. If you only need to check for specific content (like using .Contains()), you can read line-by-line or in chunks, and stop processing as soon as you find what you're looking for. This cuts down on memory drastically, especially for large responses.

Here's a revised version of your code with these fixes:

try
{
    using (HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url))
    {
        request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
        request.Headers.Add("Upgrade-Insecure-Requests", @"1");
        request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";

        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        using (StreamReader reader = new StreamReader(response.GetResponseStream()))
        {
            string line;
            // Process one line at a time instead of loading everything
            while ((line = reader.ReadLine()) != null)
            {
                if (line.Contains("your-target-content-1"))
                {
                    // Handle this case
                    break; // Exit early if you don't need to check further
                }
                else if (line.Contains("your-target-content-2"))
                {
                    // Handle another case
                }
                // Add other checks as needed
            }
        }
    }
}
catch (WebException e)
{
    // Pro tip: Log exceptions here instead of swallowing them—you'll thank yourself later
}
catch (UriFormatException p)
{
    // Same here: log the invalid URI to debug issues faster
}

3. Bonus: Consider Switching to HttpClient #

If you're working with .NET Framework 4.5+ or .NET Core/.NET 5+, HttpClient is the modern, recommended alternative to HttpWebRequest. It's designed for high-concurrency scenarios, manages connection pooling automatically, and has better memory efficiency overall. It would simplify your code too!

About Switch Statements vs. Multiple ifs #

You're right to suspect this isn't causing your memory issue—switch statements don't affect memory usage much, but they can improve readability and performance in some cases. That said:

• Switch works great if you're checking against fixed constant values (e.g., if (status == "OK") → switch (status) { case "OK": ... }). The compiler optimizes switch statements for constant checks, making them faster than a long chain of ifs.
• Switch won't help with .Contains() checks—since you're looking for substrings, not exact matches, you can't use a switch here. You'll still need if statements, or you could refactor the checks into a list of delegates for cleaner code.

For example, if your checks are all .Contains() calls, you could do something like this to clean up the code without a switch:

var checks = new List<(string Content, Action Handler)>
{
    ("content-1", () => { /* Handle case 1 */ }),
    ("content-2", () => { /* Handle case 2 */ }),
    // Add more checks here
};

foreach (var check in checks)
{
    if (line.Contains(check.Content))
    {
        check.Handler();
        break;
    }
}

This keeps your code organized without relying on switch.

内容的提问来源于stack exchange，提问作者James D