Azure应用/函数中HttpClient.GetAsync调用抛出异常求助

我在Azure Function中编写了如下代码：

[Function("GetHtml")]
public async Task GetHtml(
    [HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequest req)
{
    try
    {
        var url = $"https://www.stepstone.de/jobs/ausbildung/in-deutschland?page=1";
        var httpClient = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
        httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36");
        var response = await httpClient.GetAsync(url);
        var streamResponse = await response.Content.ReadAsStringAsync();
    }
    catch (Exception ex)
    {
        _logger.LogError(ex.Message);
    }
}

本地运行一切正常，但发布到Azure后，执行httpClient.GetAsync(url)时出现错误：

请求发送时出错。无法从传输连接读取数据：远程主机强制关闭了一个现有的连接。
[Error] 请求发送时出错。

对应的堆栈跟踪：

StackTrace: at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at AzureHtmlTest.FunctionApp.Function1.GetHtml(HttpRequest req) in C:\10_ProjectFiles\Fujour\2024.09.05

奇怪的是，将URL改为https://www.google.com/search?q=Ausbildung后，本地和Azure上都能正常运行。

我已尝试以下方法但均无效：

• 发布到Azure App（无变化）
• 设置ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11;
• 设置httpClient.DefaultRequestHeaders.Add("ContentType", "text/html; charset=utf-8");
• 设置httpClient.DefaultRequestHeaders.Add("Timeout", "30000");

恳请帮忙解决！

解决方案 #

1. 复用HttpClient实例 #

每次请求新建HttpClient会导致端口耗尽，这是Azure环境中常见的连接问题根源。改为静态单例或通过依赖注入复用：

private static readonly HttpClient _httpClient = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true });
private readonly ILogger _logger;

static Function1()
{
    // 模拟完整浏览器请求头
    _httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36");
    _httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    _httpClient.DefaultRequestHeaders.Add("Accept-Language", "de-DE,de;q=0.8,en-US;q=0.5,en;q=0.3");
    _httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
    _httpClient.DefaultRequestHeaders.Add("Connection", "keep-alive");
    // 显式设置TLS版本
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;
}

public Function1(ILogger<Function1> logger)
{
    _logger = logger;
}

[Function("GetHtml")]
public async Task GetHtml(
    [HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequest req)
{
    try
    {
        var url = "https://www.stepstone.de/jobs/ausbildung/in-deutschland?page=1";
        var response = await _httpClient.GetAsync(url);
        response.EnsureSuccessStatusCode(); // 触发HTTP错误码异常便于排查
        var streamResponse = await response.Content.ReadAsStringAsync();
        // 处理响应内容
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "请求stepstone.de接口失败");
    }
}

2. 补充完整请求头 #

stepstone.de可能会校验请求头的完整性，仅设置User-Agent不足以通过反爬校验，添加浏览器常用的Accept、Accept-Language等头可以模拟真实用户请求。

3. 检查Azure网络配置 #

• 如果使用Premium/Dedicated计划：检查网络安全组（NSG）的出站规则，确保允许访问stepstone.de的443端口。
• 如果使用Consumption计划：确认Azure Function的出站网络未被限制，可通过Kudu工具执行telnet stepstone.de 443测试连通性。

4. 尝试VNET集成（可选） #

如果stepstone.de拦截了Azure公共IP段，可以将Function集成到VNET中，使用VNET的出站IP访问目标站点，避免被IP封禁。

内容的提问来源于stack exchange，提问作者Santosh Tatpati