Prometheus Operator Helm部署后Ingress配置异常求助

Let’s work through your two issues step by step—they’re both tied to mismatched path/URL configurations between the Ingress and the services themselves.

1. Prometheus Ingress Returning 404 Not Found #

Your current Ingress routes requests to /prometheus to the Prometheus service, but Prometheus is hardcoded to serve content from the root path (/) by default. That’s why you’re getting a 404—Prometheus doesn’t recognize the /prometheus prefix. Here are two fixes:

Option 1: Update Prometheus to Use the /prometheus Route Prefix #

Since you installed the chart without modifying values.yaml, you can adjust Prometheus’s configuration via a helm upgrade command:

helm upgrade my-prom-check prometheus-community/kube-prometheus-stack -n monitoring \
  --set prometheus.prometheusSpec.routePrefix="/prometheus"

(Replace my-prom-check with your actual Helm release name—find it with helm list -n monitoring)

After running this, restart the Prometheus pod to apply the change, and your existing Ingress should start working.

Option 2: Use NGINX Ingress Rewrite Rules (If You’re Using NGINX Controller) #

If you don’t want to modify Prometheus’s core config, adjust your Ingress to rewrite the path before sending it to the service. Update your Ingress YAML like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prom
  namespace: monitoring
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: abc.xyz.com
    http:
      paths:
      - backend:
          serviceName: my-prom-check-prometheus-o-prometheus
          servicePort: 9090
        path: /prometheus(/|$)(.*)

Also, double-check that serviceName matches the actual Prometheus service in your monitoring namespace. Run kubectl get svc -n monitoring to confirm—valid service names usually look like <release-name>-prometheus.

2. Grafana Returning Invalid Key Error #

This error almost always stems from Grafana’s root_url not matching the actual URL you use to access it via Ingress. Grafana uses this setting to generate valid session cookies; if it’s wrong, you’ll get the "invalid key" message.

Fix it by updating Grafana’s config via Helm:

helm upgrade my-prom-check prometheus-community/kube-prometheus-stack -n monitoring \
  --set grafana.grafana.ini.server.root_url="http://abc.xyz.com/grafana"

(Adjust the protocol to https if you’re using TLS, and replace /grafana with your actual Grafana Ingress path)

If your Grafana Ingress uses a subpath like /grafana, you can also add --set grafana.grafana.ini.server.serve_from_sub_path: true to make Grafana natively serve content from that subpath instead of relying on rewrites.

内容的提问来源于stack exchange，提问作者Arpan Sharma