Azure容器中代理后的NodeJS连接提前关闭问题排查

问题现象 #

在Azure容器应用部署了Nginx反向代理+NodeJS API两个容器，NodeJS通过express.static挂载Azure文件共享，提供JSON数据和约400kb的小图片。本地部署正常，但线上环境出现以下异常：

• JSON请求（最大200kb）可正常处理
• 图片仅能部分下载（加载进度50%-80%），浏览器报错200 ERR_CONTENT_LENGTH_MISMATCH
• Nginx日志显示上游连接提前关闭，日志内容：

2023/12/04 17:36:02 [error] 107#107: *126 upstream prematurely closed connection while
reading upstream,
client: 100.100.0.48,
server: ,
request: "GET /tools/evidence/api/content/d8e5a7fd-2330-42ca-949f-b4a5bc5b3dcc.png 
HTTP/1.1",
upstream: "http://100.100.154.166:5000/content/d8e5a7fd-2330-42ca-949f-b4a5bc5b3dcc.png", 
host: "s175d01-ca-evidence-www.bravesky-af1cf594.westeurope.azurecontainerapps.io:443"

已尝试的排查动作 #

• 扩容容器，无效果
• 调整NodeJS服务超时参数：

server.setTimeout(1000)
server.keepAliveTimeout = 65000;

• 排除配置限制类问题，无效果

相关配置代码 #

Nginx配置 #

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=2g inactive=60m use_temp_path=off;
proxy_cache_valid 200 302 60m;

server {

    listen 80 default_server;

    # Some security headers
    server_tokens off; # Turns of version reporting in error messages and headers
    more_clear_headers "Server"; # Clears the Server header containing server info
    more_clear_headers "X-Powered-By"; # Clears the X-Powered-By header containing server info
    more_set_headers "Referrer-Policy: strict-origin-when-cross-origin"; # Make the default explicit
    more_set_headers "X-Content-Type-Options: nosniff"; # Blocks requests from script tags if not Javascript and styles tags if not CSS

    # Enable compression
    gzip on;
    gzip_min_length 20; # Small files offer little benefit from compression
    gzip_disable "msie6"; 
    gzip_vary on; # Lets the client know through the Vary header that different encodings will affect cache
    gzip_proxied any; # Enable compression for all proxied requests
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml
      application/xml+rss text/javascript application/vnd.ms-fontobject application/x-font-opentype 
      application/x-font-truetype application/x-font-ttf font/eot font/opentype font/otf image/svg+xml 
      image/vnd.microsoft.icon;

    add_header X-Cache-Status $upstream_cache_status;
    proxy_buffering on;

    # A health check endpoint
    location = /health {
        return 204;
        access_log "off"; # Don't log
    }

    # Needs to be / or /base/path/
    location ${BASE_URL} {   
        auth_basic "Protected area";
        auth_basic_user_file /etc/nginx/users.htpasswd;
        index index.html;
        alias /srv/www/;
        # Needs to be / or /base/path/
        try_files $uri $uri/ ${BASE_URL}index.html;
    }

    # Needs to be / or /base/path/
    location ^~ ${BASE_URL}api/ {
        auth_basic "Protected area";
        auth_basic_user_file /etc/nginx/users.htpasswd;
        proxy_pass ${PROXY_API_BASE_URL}/;
    }

}

NodeJS核心代码 #

const main = async () => {
  await repository.init();

  const app = express();

  // Support JSON encoded requests
  app.use(express.json());

  // GZip responses
  app.use(compression());

  // Enable Cross-site Origin Requests for all domains in development mode. In production we would expose
  // the API and the web app on the same domain through the reverse-proxy.
  if (config.NODE_ENV === "development") {
    app.use(cors());
  }

  app.use("/content", express.static(config.IMG_LOCAL_PATH));

  const resolvers = {
    // ... resolver definitions
  };

  const apollo = new ApolloServer({
    typeDefs,
    resolvers,
  });

  // Must await the start of Apollo, before adding as middleware
  await apollo.start();
  app.use("/graphql", expressMiddleware(apollo));

  if (config.LOG_REQUESTS) {
    app.use((req, res, next) => {
      const timestamp = new Date().toLocaleString();
      logger.log(`${timestamp}|${req.method}|${req.url}`);
      next();
    });
  }

  const server = app.listen(config.API_PORT, () => {
    logger.log(`NODE_ENV is: ${config.NODE_ENV}`);
    logger.log(`API listening on port ${config.API_PORT}`);
  });

  server.setTimeout(1000)
  server.keepAliveTimeout = 65000;

};

main();

怀疑方向 #

怀疑Azure文件共享挂载存在异常，但NodeJS日志未输出相关错误信息，恳请帮忙排查问题原因及解决办法。

内容的提问来源于stack exchange，提问作者Brendan