如何通过ASP.NET C#程序化修改网站语言并实现数据抓取

Let's break down how to handle language switching during web scraping with ASP.NET C#—and use the Tamil Nadu PDS portal (https://www.tnpds.gov.in) as a concrete example to walk through.

1. General Approach to Switching Language & Fetching Responses #

Most websites handle language switching in one of a few common ways:

• POST form submission: Sending a form parameter that specifies the desired language (like your example)
• GET query parameter: Appending something like ?lang=en directly to the URL
• Cookie: Storing the language preference in a cookie the server checks on subsequent requests
• Accept-Language HTTP header: Sending a header like Accept-Language: en-US to request content in a specific language

For scraping, you’ll first need to inspect the website’s language switcher (using browser DevTools) to figure out which method it uses. For tnpds.gov.in, the site relies on a POST form submission, so we’ll focus on that approach here.

2. Example Code for https://www.tnpds.gov.in #

Here’s a polished version of your code with explanations for each step, plus best practices like proper resource management:

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Text;
using System.Linq;
using HtmlAgilityPack; // Don't forget to install this NuGet package

class LanguageSwitchScraper
{
    static void Main()
    {
        string targetUrl = "https://www.tnpds.gov.in";
        string desiredLanguage = "en"; // Swap to "ta" for Tamil, etc.

        // Initialize the POST request
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(targetUrl);
        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        // Mimic a real browser to avoid being blocked
        request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36";
        request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

        // Build form parameters (match the site's actual field names exactly)
        var postParams = new Dictionary<string, string>
        {
            {"masterForm:languageSelectMenu:", desiredLanguage}
        };

        // Encode parameters into the request body
        string postData = string.Join("&", postParams.Select(kv => $"{Uri.EscapeDataString(kv.Key)}={Uri.EscapeDataString(kv.Value)}"));
        byte[] byteArray = Encoding.UTF8.GetBytes(postData);
        request.ContentLength = byteArray.Length;

        // Write the post data to the request stream
        using (Stream dataStream = request.GetRequestStream())
        {
            dataStream.Write(byteArray, 0, byteArray.Length);
        }

        // Fetch and process the response
        try
        {
            using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            {
                if (response.StatusCode == HttpStatusCode.OK)
                {
                    // Read content with the correct encoding from the server
                    using (Stream receiveStream = response.GetResponseStream())
                    using (StreamReader readStream = response.CharacterSet == null 
                        ? new StreamReader(receiveStream) 
                        : new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet)))
                    {
                        string htmlContent = readStream.ReadToEnd();
                        
                        // Load into HtmlAgilityPack for parsing (if needed)
                        HtmlDocument htmlDoc = new HtmlDocument();
                        htmlDoc.LoadHtml(htmlContent);

                        // Add your parsing logic here—e.g., extract specific elements
                        Console.WriteLine($"Successfully fetched content in {desiredLanguage}!");
                    }
                }
            }
        }
        catch (WebException ex)
        {
            // Handle common errors like timeouts or blocked requests
            Console.WriteLine($"Error fetching content: {ex.Message}");
            if (ex.Response != null)
            {
                using (HttpWebResponse errorResponse = (HttpWebResponse)ex.Response)
                {
                    Console.WriteLine($"Server returned status code: {errorResponse.StatusCode}");
                }
            }
        }
    }
}

Key Details to Note: #

• Form Parameter Name: The masterForm:languageSelectMenu: key comes directly from the site’s language dropdown form. You can find this by right-clicking the language selector in your browser → Inspect Element.
• User Agent: Setting a realistic user agent helps avoid being flagged as a bot by the website’s anti-scraping measures.
• Resource Management: Using using statements ensures streams and responses are properly disposed of, preventing memory leaks.
• Error Handling: The try/catch block covers common issues like connection failures or server errors, making your scraper more robust.

内容的提问来源于stack exchange，提问作者Prasanna