配置Nginx反向代理后Grafana与Prometheus无法加载排查求助

我在Ubuntu系统用Docker部署了Grafana（3000端口）和Prometheus（9090端口），原本能通过IP+端口的HTTP方式正常访问。配置Nginx做反向代理实现HTTPS加密后，这两个服务无法通过加密URL加载，求排查解决。

运行Prometheus的Docker命令 #

sudo docker run -d --user 998:998 --net=host -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /data/prometheus:/data/prometheus prom/prometheus --config.file="/etc/prometheus/prometheus.yml" --storage.tsdb.path="/data/prometheus"

当前Nginx配置 #

server {
    server_name www.mydomain.com;
    root /var/www/html;

    #server_tokens off;   
    index index.html index.htm;        
    location / {         
        # First attempt to server request as file, then         
        # as directory, then fall back to displaying a 404.           
        try_files $uri $uri/ /index.html =404;      
    }

    location /prometheus {
                proxy_pass http://localhost:9090;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_cache_bypass $http_upgrade;

    }
    location /grafana {
                proxy_pass http://localhost:3000;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_cache_bypass $http_upgrade;
    }
    location /node_exporter {
                proxy_pass http://localhost:9100;
    }

    # Access and error logs
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
}

排查与解决步骤 #

1. 修正Nginx反向代理路径映射 #

当前配置中location /prometheus和location /grafana的proxy_pass末尾未加斜杠，会导致请求路径拼接错误。比如访问https://www.mydomain.com/prometheus时，Nginx会转发到http://localhost:9090/prometheus，但Prometheus的根路径是/，因此无法匹配资源。

修改后的Nginx配置片段：

location /prometheus/ {
    proxy_pass http://localhost:9090/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_cache_bypass $http_upgrade;
}

location /grafana/ {
    proxy_pass http://localhost:3000/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_cache_bypass $http_upgrade;
}

注意：location和proxy_pass末尾都要添加斜杠，确保路径正确映射。

2. 配置Grafana的子路径访问 #

Grafana需要明确自己被代理的路径，否则静态资源、API请求会走错误路径。

• 若挂载了Grafana配置文件，修改grafana.ini：

[server]
root_url = https://www.mydomain.com/grafana/
serve_from_sub_path = true

• 若用Docker环境变量传递配置，重启Grafana容器时添加参数：

sudo docker run -d --net=host -e GF_SERVER_ROOT_URL=https://www.mydomain.com/grafana/ -e GF_SERVER_SERVE_FROM_SUB_PATH=true grafana/grafana

3. 配置Prometheus的外部访问URL #

Prometheus需要设置外部URL，避免页面内链接跳转错误。修改prometheus.yml添加：

global:
  external_labels:
    monitor: 'my-monitor'
external_url: 'https://www.mydomain.com/prometheus/'
# 其余原有配置保留

修改后重启Prometheus容器：

sudo docker restart <prometheus-container-id>

4. 补全Nginx的HTTPS基础配置 #

确保Nginx配置包含SSL证书相关设置，否则HTTPS访问本身会失败：

server {
    listen 443 ssl;
    server_name www.mydomain.com;
    ssl_certificate /path/to/your/cert.pem;
    ssl_certificate_key /path/to/your/key.pem;
    
    # 原有location配置...
}

# 可选：将HTTP请求强制重定向到HTTPS
server {
    listen 80;
    server_name www.mydomain.com;
    return 301 https://$host$request_uri;
}

5. 验证配置并重启服务 #

• 检查Nginx配置语法：

sudo nginx -t

• 重载Nginx配置：

sudo nginx -s reload

• 重启Grafana和Prometheus容器，确保新配置生效。

6. 查看日志定位问题 #

若仍无法访问，检查以下日志获取具体错误信息：

• Nginx错误日志：/var/log/nginx/error.log
• Grafana日志：sudo docker logs <grafana-container-id>
• Prometheus日志：sudo docker logs <prometheus-container-id>

内容的提问来源于stack exchange，提问作者Musharraf Shaikh