如何在Kubernetes中实现TCP协议的SSL终止并部署带外部SSL终止的MQTT Broker

Great question! Let's walk through how to set up SSL termination for your MQTT broker (listening on TCP 1883 internally) using both HAProxy Ingress and Nginx Ingress, plus cover the core ideas behind TCP-layer SSL termination in Kubernetes.

Unlike HTTP traffic (which uses standard Ingress resources), TCP traffic requires Ingress controllers to directly listen on specific ports, terminate the SSL/TLS handshake, then forward the unencrypted TCP stream to your backend service (your MQTT broker on port 1883). The key steps are:

• Store your SSL certificate as a Kubernetes Secret.
• Configure the Ingress controller to listen on an external TLS port (typically 8883 for MQTT over TLS).
• Map that external port to your internal MQTT broker service, enabling SSL termination at the Ingress layer.

HAProxy Ingress Controller: Step-by-Step Setup #

HAProxy has native support for TCP SSL termination with minimal configuration.

1. Create a TLS Secret #

First, store your SSL certificate and private key in a Kubernetes Secret (replace paths and names as needed):

kubectl create secret tls mqtt-tls-secret \
  --cert=./fullchain.pem \
  --key=./privkey.pem \
  -n your-broker-namespace

2. Update HAProxy's ConfigMap #

Edit the HAProxy Ingress ConfigMap (usually in the kube-system namespace) to define the TCP port mapping and enable SSL termination:

apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-ingress
  namespace: kube-system
data:
  # Define TCP service mapping: external_port=namespace/service:internal_port,tls
  tcp-services: |
    8883=your-broker-namespace/mqtt-broker-service:1883,tls
  # Specify the TLS secret to use for termination
  ssl-certificate: kube-system/mqtt-tls-secret

The ,tls suffix tells HAProxy to terminate SSL on this port before forwarding traffic to the backend.

3. Verify and Test #

• Check the HAProxy Ingress pod logs to confirm the configuration was applied:

  kubectl logs -n kube-system <haproxy-ingress-pod-name>
  

• Use an MQTT client (like mosquitto_pub) to test the connection:

  mosquitto_pub -h your-ingress-ip -p 8883 -t "test/topic" -m "hello" --cafile ./ca.pem
  

Nginx Ingress Controller: Step-by-Step Setup #

Nginx requires a few more steps, as TCP configuration is handled separately from standard HTTP Ingress resources.

1. Create a TLS Secret #

Same as with HAProxy, create the TLS Secret for your certificate:

kubectl create secret tls mqtt-tls-secret \
  --cert=./fullchain.pem \
  --key=./privkey.pem \
  -n your-broker-namespace

2. Configure TCP Services ConfigMap #

Create or update a dedicated ConfigMap for TCP services (usually in kube-system):

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-tcp-services
  namespace: kube-system
data:
  # Map external port 8883 to internal broker service
  8883: "your-broker-namespace/mqtt-broker-service:1883"

3. Update Nginx Ingress Deployment #

Modify the Nginx Ingress Deployment to:

• Expose the 8883 port on the pod/host
• Mount the TLS Secret so Nginx can access the certificate

Add these sections to the Deployment spec:

spec:
  template:
    spec:
      containers:
      - name: nginx-ingress-controller
        ports:
        - name: mqtt-tls
          containerPort: 8883
          hostPort: 8883
        volumeMounts:
        - name: mqtt-tls-cert
          mountPath: /etc/nginx/secrets/mqtt-tls
          readOnly: true
      volumes:
      - name: mqtt-tls-cert
        secret:
          secretName: mqtt-tls-secret
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key

4. Update TCP ConfigMap to Use TLS #

Go back to the nginx-ingress-tcp-services ConfigMap and update the port mapping to reference the mounted certificate:

data:
  8883: "your-broker-namespace/mqtt-broker-service:1883,cert=/etc/nginx/secrets/mqtt-tls/tls.crt,key=/etc/nginx/secrets/mqtt-tls/tls.key"

5. Test the Connection #

Use the same mosquitto_pub command as before to verify the encrypted connection works:

mosquitto_pub -h your-ingress-ip -p 8883 -t "test/topic" -m "hello" --cafile ./ca.pem

• Use Standard Ports: Stick to 8883 for MQTT over TLS (industry standard) to avoid confusion.
• Automate Certificates: Use Cert-Manager to automatically issue and renew Let's Encrypt certificates, so you don't have to manually manage them.
• Restrict Access: Add IP whitelists to your Ingress controller (HAProxy uses haproxy.org/whitelist-source-range, Nginx uses nginx.ingress.kubernetes.io/whitelist-source-range) to limit who can connect to your MQTT broker.
• Monitor Traffic: Enable logging in your Ingress controller to track TCP connections, SSL handshakes, and any errors.

内容的提问来源于stack exchange，提问作者William