C#中HttpWebResponse返回ASCII字符而非VirusTotal网页HTML的问题排查

Alright, let's figure out why you're getting those weird black square characters and how to fix this issue:

Why the weird black squares? #

There are two main reasons behind this problem, with the first being the most likely culprit:

• Compressed response content: VirusTotal automatically returns Gzip/Deflate compressed content to reduce bandwidth usage. Your current code doesn't handle decompression, so when you read the raw compressed stream as text, it shows up as unreadable black squares. Other sites like Google/Microsoft might not enforce compression for your requests, which is why they work fine.
• Anti-scraping measures: VirusTotal detects non-browser requests (your code doesn't send browser-like headers like User-Agent), which might trigger a blocked or malformed response. However, since you're getting some content instead of an error code, compression is the more immediate issue.

Fixes to resolve this #

1. Handle compressed content in your code #

Modify your existing code to check for and decompress the response. You'll need to add support for Gzip/Deflate streams:

using System.IO.Compression; // Don't forget this namespace!

// ...

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.virustotal.com/#/domain/" + domain);
// Add a browser-like User-Agent to avoid anti-scraping flags
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36";
// Explicitly tell the server we accept compressed content
request.AcceptEncoding = "gzip, deflate";

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string page = string.Empty;

if (response.StatusCode == HttpStatusCode.OK)
{
    Stream receiveStream = response.GetResponseStream();

    // Decompress if needed
    if (response.ContentEncoding.Equals("gzip", StringComparison.OrdinalIgnoreCase))
    {
        receiveStream = new GZipStream(receiveStream, CompressionMode.Decompress);
    }
    else if (response.ContentEncoding.Equals("deflate", StringComparison.OrdinalIgnoreCase))
    {
        receiveStream = new DeflateStream(receiveStream, CompressionMode.Decompress);
    }

    StreamReader readStream = response.CharacterSet == null 
        ? new StreamReader(receiveStream) 
        : new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
    
    page = readStream.ReadToEnd();
    
    // Clean up resources properly
    readStream.Dispose();
    response.Dispose();
    MessageBox.Show(page);
}

2. Use VirusTotal's Official API (Strongly Recommended) #

Scraping VirusTotal's web pages violates their Terms of Service, and web page structures change frequently—this will break your code unexpectedly. Instead, use their free public API, which is designed for exactly this kind of automated querying:

1. Sign up for a free VirusTotal account to get an API key.
2. Use their Domain Report endpoint to get structured JSON data (no messy HTML parsing needed):

string apiKey = "YOUR_PERSONAL_API_KEY"; // Replace with your actual key
string domain = "example.com"; // Your target domain

HttpWebRequest request = (HttpWebRequest)WebRequest.Create($"https://www.virustotal.com/api/v3/domains/{domain}");
request.Headers.Add("x-apikey", apiKey);
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36";

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader readStream = new StreamReader(response.GetResponseStream());
string jsonResponse = readStream.ReadToEnd();

// Parse the JSON to extract domain details (source, status, existence, etc.)
MessageBox.Show(jsonResponse);

// Clean up resources
readStream.Dispose();
response.Dispose();

This method is reliable, compliant, and gives you structured data that's easy to parse.

内容的提问来源于stack exchange，提问作者Tom Hood