使用C#访问带登录验证网站的请求/响应头与请求体问题

Hey there! I see you're trying to pull data from a password-protected site via proxy, and you're stuck on capturing full request/response details (headers + bodies) instead of just the response content. Let's break down what's going on and fix this.

The Problem with Your Current Code #

Your mix of HttpWebRequest and WebClient is holding you back:

• WebClient is a simplified wrapper that hides low-level details like the request headers you send, and it doesn't make logging the full request body easy.
• You're not persisting session cookies between requests, which might even break your authenticated state—login sessions rely on cookies to stay active, after all.

Solution: Use HttpWebRequest for Full Control #

HttpWebRequest gives you complete access to both request and response headers, plus the ability to capture request/response bodies. We'll also add a CookieContainer to maintain your login session across requests, and configure the proxy properly.

Here's a revised version of your code that captures all the details you need:

using System;
using System.IO;
using System.Net;
using System.Text;

namespace ScrapeTest
{
    class Program
    {
        static void Main(string[] args)
        {
            // Configure proxy settings once for all requests
            var proxy = new WebProxy("proxyServer", true)
            {
                Credentials = new NetworkCredential("proxyUsername", "proxyPassword"),
                UseDefaultCredentials = false // Avoid overriding our explicit proxy credentials
            };

            // Cookie container to persist login session across requests
            var cookieContainer = new CookieContainer();

            // 1. Initial authenticated request to establish session
            var baseRequest = (HttpWebRequest)WebRequest.Create("remoteserver");
            baseRequest.Timeout = 50000;
            baseRequest.MaximumAutomaticRedirections = 7;
            baseRequest.Credentials = new NetworkCredential("username", "password");
            baseRequest.Proxy = proxy;
            baseRequest.CookieContainer = cookieContainer;

            // Log outgoing request headers
            Console.WriteLine("=== Initial Request Headers ===");
            foreach (string headerKey in baseRequest.Headers)
            {
                Console.WriteLine($"{headerKey}: {baseRequest.Headers[headerKey]}");
            }

            using (var baseResponse = (HttpWebResponse)baseRequest.GetResponse())
            {
                // Log incoming response headers
                Console.WriteLine("\n=== Initial Response Headers ===");
                foreach (string headerKey in baseResponse.Headers)
                {
                    Console.WriteLine($"{headerKey}: {baseResponse.Headers[headerKey]}");
                }

                // Read and log response body
                using (var reader = new StreamReader(baseResponse.GetResponseStream(), Encoding.UTF8))
                {
                    string responseBody = reader.ReadToEnd();
                    Console.WriteLine("\n=== Initial Response Body ===");
                    Console.WriteLine(responseBody);
                }
            }

            // 2. Request data from otherPage using the same authenticated session
            var dataRequest = (HttpWebRequest)WebRequest.Create("otherPage");
            dataRequest.Method = "GET";
            dataRequest.Proxy = proxy;
            dataRequest.CookieContainer = cookieContainer; // Reuse session cookies
            dataRequest.Credentials = new NetworkCredential("username", "password"); // Re-authenticate if required

            // Log outgoing request headers for this second request
            Console.WriteLine("\n=== Data Request Headers ===");
            foreach (string headerKey in dataRequest.Headers)
            {
                Console.WriteLine($"{headerKey}: {dataRequest.Headers[headerKey]}");
            }

            using (var dataResponse = (HttpWebResponse)dataRequest.GetResponse())
            {
                // Log incoming response headers
                Console.WriteLine("\n=== Data Response Headers ===");
                foreach (string headerKey in dataResponse.Headers)
                {
                    Console.WriteLine($"{headerKey}: {dataResponse.Headers[headerKey]}");
                }

                // Read and log response body
                using (var reader = new StreamReader(dataResponse.GetResponseStream(), Encoding.UTF8))
                {
                    string responseBody = reader.ReadToEnd();
                    Console.WriteLine("\n=== Data Response Body ===");
                    Console.WriteLine(responseBody);
                }
            }

            Console.ReadLine();
        }
    }
}

Key Improvements Explained #

• CookieContainer: This tracks cookies set by the server after login, so your subsequent requests are recognized as authenticated.
• Full Header Logging: We loop through request.Headers and response.Headers to print every header sent and received.
• Consistent Request Setup: Using HttpWebRequest for all requests ensures you have access to all low-level details, unlike the limited WebClient.
• Proxy Configuration: We set the proxy once and reuse it across requests to avoid redundant setup.

Bonus: Capturing Request Body (For POST Requests) #

If you need to send a POST request and log the request body, add this before calling GetResponse():

string postData = "key1=value1&key2=value2";
byte[] postBytes = Encoding.UTF8.GetBytes(postData);

dataRequest.Method = "POST";
dataRequest.ContentType = "application/x-www-form-urlencoded";
dataRequest.ContentLength = postBytes.Length;

// Log request body
Console.WriteLine("\n=== Request Body ===");
Console.WriteLine(postData);

// Write body to the request stream
using (var requestStream = dataRequest.GetRequestStream())
{
    requestStream.Write(postBytes, 0, postBytes.Length);
}

That should give you full visibility into every part of your requests and responses!

内容的提问来源于stack exchange，提问作者Josh Dow