如何通过VPN隧道用Fluent-Bit将本地K8s日志发送至GCP并查看

一、前提确认 #

• 本地4节点K8s集群与GCP子网通过Mikrotik VPN隧道双向通信正常
• 已启用GCP侧Cloud Logging和Cloud Storage API
• 已创建目标GCP Bucket，且拥有具备roles/logging.logWriter、roles/storage.objectCreator权限的服务账号，密钥已下载至本地

二、部署Fluent-Bit到本地K8s集群 #

以DaemonSet方式部署Fluent-Bit，确保每个节点采集容器日志。

1. 创建K8s Secret存储GCP服务账号密钥 #

kubectl create secret generic fluent-bit-gcp-key --from-file=key.json=/path/to/your/service-account-key.json -n kube-system

2. 编写Fluent-Bit配置ConfigMap #

核心配置采集源、K8s元数据过滤，同时输出到Cloud Logging和Cloud Storage，强制流量走VPN：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
        Daemon       off
        HTTP_Server  on
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        IPv4_Filter  on # 强制使用IPv4，避免解析到公网IP

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name              kubernetes
        Match             kube.*
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name              google_cloud
        Match             *
        Host              logging.googleapis.com
        Project_ID        your-gcp-project-id
        Service_Account_Key /fluent-bit/etc/key.json
        Log_ID            your-log-bucket-id # Log Viewer对应的日志桶ID
        Storage_Bucket    gs://your-target-bucket-name # 目标Bucket地址
        Storage_Path      k8s-logs/%Y/%m/%d # Bucket内日志存储路径
        Storage_Sync_Interval 60

  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

3. 部署Fluent-Bit DaemonSet #

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit
        image: cr.fluentbit.io/fluent/fluent-bit:2.2.0
        ports:
        - containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: fluent-bit-gcp-key
          mountPath: /fluent-bit/etc/key.json
          subPath: key.json
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      - name: fluent-bit-gcp-key
        secret:
          secretName: fluent-bit-gcp-key
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

三、强制日志流量通过VPN隧道 #

1. GCP侧配置 #

启用VPC的Private Google Access，允许子网资源访问Google服务私有IP；记录GCP服务私有IP段：

• Logging：199.36.153.4/30
• Storage：199.36.153.8/30

2. Mikrotik路由器配置 #

添加静态路由，将上述IP段指向VPN隧道出口，确保本地K8s节点访问这些IP时走VPN。

3. K8s节点路由配置 #

在每个本地节点添加静态路由，强制流量指向Mikrotik内网IP：

ip route add 199.36.153.4/30 via <mikrotik-internal-ip> dev <local-network-interface>
ip route add 199.36.153.8/30 via <mikrotik-internal-ip> dev <local-network-interface>

四、验证配置 #

1. 检查Fluent-Bit Pod状态：

kubectl get pods -n kube-system -l k8s-app=fluent-bit-logging

2. 查看Pod日志确认无连接错误：

kubectl logs -n kube-system <fluent-bit-pod-name>

3. 登录GCP Console，检查Log Viewer是否有本地K8s日志，同时确认目标Bucket生成日志文件
4. 本地节点抓包验证流量走VPN隧道，而非公网

五、额外优化 #

• 配置Fluent-Bit过滤规则，仅发送必要日志以减少流量
• 启用缓冲功能，避免VPN中断时丢失日志
• 定期轮换服务账号密钥，提升安全性

内容的提问来源于stack exchange，提问作者RedBluff