如何用Telegraf从Kafka指标端点采集JMX指标？TICK栈下无需Jolokia插件导入InfluxDB可行吗？

Great question—yes, you absolutely don't need the Jolokia Telegraf plugin to get Kafka metrics into InfluxDB when running on Kubernetes. There are two reliable, straightforward methods to do this, and I'll walk you through both, plus how to set up Telegraf to collect Kafka's JMX metrics without relying on Jolokia.

1. Can You Skip Jolokia? Short Answer: Yes #

Jolokia is just one tool to access JMX metrics over HTTP, but Kafka gives you better alternatives that don't require it:

• Method 1: Use Telegraf's dedicated kafka_jmx input plugin to connect directly to Kafka's native JMX port (no HTTP proxy needed)
• Method 2: Deploy Kafka's official JMX Exporter as a sidecar to expose metrics via a Prometheus-compatible HTTP endpoint, then scrape it with Telegraf's prometheus input plugin

Both methods work seamlessly in Kubernetes and let you avoid Jolokia entirely.

2. How to Collect Kafka JMX Metrics with Telegraf (No Jolokia) #

Let's break down each approach with Kubernetes-specific configurations:

Method 1: Use Telegraf's kafka_jmx Plugin (Direct JMX Connection) #

This plugin is purpose-built for Kafka and connects directly to Kafka's JMX port, so no extra proxy tools like Jolokia are required.

Step 1: Configure Kafka to Expose JMX Port

Update your Kafka StatefulSet/Deployment manifest to enable JMX and expose the port to the cluster:

spec:
  template:
    spec:
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:latest
        env:
          - name: KAFKA_JMX_PORT
            value: "9999"
          - name: KAFKA_JMX_HOSTNAME
            value: "0.0.0.0" # Allow connections from any pod in the cluster
        ports:
          - name: jmx
            containerPort: 9999
            protocol: TCP

Step 2: Configure Telegraf's kafka_jmx Input Plugin

Add this configuration to your Telegraf ConfigMap (or inline config) to target Kafka's JMX service:

[[inputs.kafka_jmx]]
  ## Kafka JMX connection string (use your Kafka service name here)
  servers = ["service:jmx:rmi:///jndi/rmi://kafka.default.svc.cluster.local:9999/jmxrmi"]

  ## Optional: Filter specific metrics to reduce noise (leave empty to collect all)
  # metrics = ["kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec", "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions"]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb.default.svc.cluster.local:8086"]
  token = "$INFLUXDB_TOKEN"
  organization = "your-org"
  bucket = "kafka-metrics"

Step 3: Expose Kafka's JMX Port via Service

Make sure your Kafka service exposes the JMX port so Telegraf can reach it:

apiVersion: v1
kind: Service
metadata:
  name: kafka
spec:
  ports:
  - name: jmx
    port: 9999
    targetPort: 9999
  selector:
    app: kafka

Method 2: Prometheus Exporter Sidecar + Telegraf's prometheus Plugin #

If you prefer exposing metrics via HTTP (often easier for cluster-wide monitoring), use Kafka's official JMX Exporter as a sidecar, then scrape it with Telegraf.

Step 1: Add JMX Exporter Sidecar to Kafka Deployment

First, create a ConfigMap to define which metrics the exporter should expose:

apiVersion: v1
kind: ConfigMap
metadata:
  name: jmx-exporter-config
data:
  config.yaml: |
    lowercaseOutputName: true
    rules:
    - pattern: "kafka.server<type=(.+), name=(.+)><>(.+)"
      name: kafka_server_$1_$2_$3

Update your Kafka StatefulSet to include the sidecar and attach the config:

spec:
  template:
    spec:
      containers:
      - name: kafka
        # ... existing Kafka config ...
        env:
          - name: KAFKA_OPTS
            value: "-javaagent:/usr/local/bin/jmx_prometheus_javaagent.jar=9404:/etc/jmx-exporter/config.yaml"
        volumeMounts:
          - name: jmx-exporter-config
            mountPath: /etc/jmx-exporter
          - name: jmx-exporter-agent
            mountPath: /usr/local/bin
      - name: jmx-exporter
        image: quay.io/prometheus/jmx-exporter:latest
        ports:
          - name: metrics
            containerPort: 9404
        volumeMounts:
          - name: jmx-exporter-config
            mountPath: /etc/jmx-exporter
      volumes:
      - name: jmx-exporter-config
        configMap:
          name: jmx-exporter-config
      - name: jmx-exporter-agent
        emptyDir: {}

Step 2: Configure Telegraf's prometheus Input Plugin

Update your Telegraf config to scrape the exporter's HTTP endpoint:

[[inputs.prometheus]]
  ## URL of the JMX Exporter's metrics endpoint
  urls = ["http://kafka.default.svc.cluster.local:9404/metrics"]

  ## Optional: Set timeout for scrapes
  # timeout = "5s"

[[outputs.influxdb_v2]]
  urls = ["http://influxdb.default.svc.cluster.local:8086"]
  token = "$INFLUXDB_TOKEN"
  organization = "your-org"
  bucket = "kafka-metrics"

Key Kubernetes Tips #

• Network Access: Ensure Telegraf pods are in the same namespace as Kafka, or use cross-namespace service references if needed.
• Security: If your cluster uses TLS or RBAC, add necessary certificates and permissions to the Telegraf and Kafka pods.
• Resource Limits: Both the JMX Exporter and Telegraf are lightweight, but set resource requests/limits to avoid cluster resource contention.

内容的提问来源于stack exchange，提问作者Vinayak Mishra