请求纳斯达克RSS Feed在浏览器正常但代码（C#/Axios）超时

向https://www.nasdaq.com/feed/rssoutbound?category=FinTech发送HTTP GET请求时出现异常：浏览器手动访问可正常加载Feed，但通过C# HttpClient、Node.js Axios发起请求时会无限挂起最终超时，甚至使用Puppeteer抓取也会遇到同样问题。已排除Cookie影响（用Edge浏览器工具清除所有可编辑头且不带Cookie发送请求仍可正常工作），且请求其他RSS源一切正常。

C# 测试代码 #

public async Task Execute()
{
    using HttpClient httpClient = new HttpClient();
    try
    {
        // Specify the URL you want to request
        string url = "https://www.nasdaq.com/feed/rssoutbound?category=FinTech";

        httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("*/*"));
        httpClient.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en-US"));
        httpClient.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en", 0.9));
        httpClient.DefaultRequestHeaders.Add("sec-ch-ua", "\"Chromium\";v=\"116\", \"Not)A;Brand\";v=\"24\", \"Microsoft Edge\";v=\"116\"");
        httpClient.DefaultRequestHeaders.Add("sec-ch-ua-mobile", "?0");
        httpClient.DefaultRequestHeaders.Add("sec-ch-ua-platform", "\"Windows\"");
        httpClient.DefaultRequestHeaders.Add("sec-fetch-dest", "empty");
        httpClient.DefaultRequestHeaders.Add("sec-fetch-mode", "cors");
        httpClient.DefaultRequestHeaders.Add("sec-fetch-site", "same-origin");

        // Add the User-Agent header
        httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36");


        // Send a GET request and wait for the response
        HttpResponseMessage response = await httpClient.GetAsync(url);

        // Check if the request was successful
        if (response.IsSuccessStatusCode)
        {
            // Read the content of the response as a string
            string content = await response.Content.ReadAsStringAsync();

            // Print the content to the console
            Console.WriteLine(content);
        }
        else
        {
            Console.WriteLine($"HTTP request failed with status code: {response.StatusCode}");
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
        throw;
    }
}

Node.js Axios测试代码 #

// Node.js code using Axios
const axios = require('axios');

async function fetchData() {
    try {
        const response = await axios.get('https://www.nasdaq.com/feed/rssoutbound?category=FinTech', {
            headers: {
                Accept: '*/*',
                'Accept-Language': 'en-US,en;q=0.9,tr;q=0.8',
                'sec-ch-ua': '"Chromium";v="116", "Not)A;Brand";v="24", "Microsoft Edge";v="116"',
                'sec-ch-ua-mobile': '?0',
                'sec-ch-ua-platform': '"Windows"',
                'sec-fetch-dest': 'empty',
                'sec-fetch-mode': 'cors',
                'sec-fetch-site': 'same-origin',
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36'
            }
        });
        // ... (handling the response)
    } catch (error) {
        console.error('An error occurred:', error.message);
    }
}

fetchData();

可正常运行的PowerShell代码 #

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 Edg/116.0.1938.81"
Invoke-WebRequest -UseBasicParsing -Uri "https://www.nasdaq.com/feed/rssoutbound?category=FinTech" `
-WebSession $session `
-Headers @{
"authority"="www.nasdaq.com"
  "method"="GET"
  "path"="/feed/rssoutbound?category=FinTech"
  "scheme"="https"
  "accept"="*/*"
  "accept-encoding"="gzip, deflate, br"
  "accept-language"="en-US,en;q=0.9,tr;q=0.8"
  "sec-ch-ua"="`"Chromium`";v=`"116`", `"Not)A;Brand`";v=`"24`", `"Microsoft Edge`";v=`"116`""
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
  "sec-fetch-dest"="empty"
  "sec-fetch-mode"="cors"
  "sec-fetch-site"="same-origin"
}

疑问 #

请问纳斯达克Feed的此问题由何导致？其服务器配置或请求处理方式是否有特殊之处？恳请提供见解与建议。

内容的提问来源于stack exchange，提问作者Teoman shipahi