如何配置Fluentd DaemonSet同时采集控制台日志和自定义路径文件日志

配置方案总览 #

你需要同步修改应用Deployment、Fluentd DaemonSet、Fluentd采集配置三部分，额外按需配置持久卷避免日志丢失，具体操作如下：

1. 修改应用Deployment配置 #

首先需要将Pod内的自定义日志路径挂载到节点的固定目录，方便Fluentd DaemonSet读取，同时可配置持久卷挂载避免Pod崩溃日志丢失。
修改后的Deployment示例：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: carts
spec:
  replicas: 1
  selector:
    matchLabels:
      app: carts
  template:
    metadata:
      labels:
        app: carts
    spec:
      volumes:
        # 自定义日志卷，可替换为PVC实现持久化
        - name: custom-log
          hostPath:
            path: /var/log/app-custom-logs/carts # 节点上的存储路径，按服务区分
            type: DirectoryOrCreate
        # 如果需要持久化，注释上面的hostPath，用下面的PVC配置，先提前创建好carts-log-pvc
        # - name: custom-log
        #   persistentVolumeClaim:
        #     claimName: carts-log-pvc
      containers:
        - name: app
          image: carts-service
          resources:
            limits:
              memory: "1024Mi"
              cpu: "500m"
          ports:
            - containerPort: 4000
          volumeMounts:
            - name: custom-log
              mountPath: /var/log/services/dev # Pod内的日志路径

如果需要持久化，提前创建PVC（minikube默认开启hostPath类型的StorageClass，可自动绑定PV）：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: carts-log-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi # 按需调整存储大小

2. 修改Fluentd DaemonSet配置 #

给Fluentd挂载节点上存储自定义日志的路径，让Fluentd可以读取到日志文件：

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        # 新增：挂载节点自定义日志目录
        - name: app-custom-log
          hostPath:
            path: /var/log/app-custom-logs
            type: DirectoryOrCreate
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            # 新增：挂载自定义日志目录到Fluentd容器
            - name: app-custom-log
              mountPath: /var/log/app-custom-logs
              readOnly: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "elasticsearch.default"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"
            # 新增：指定自定义Fluentd配置文件路径，需要把配置存到ConfigMap挂载进来
            - name: FLUENTD_CONF
              value: fluentd.conf

3. 新增Fluentd自定义采集配置 #

创建ConfigMap存储Fluentd的自定义配置，新增自定义日志采集规则，同时保留原有控制台日志的采集逻辑：

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: kube-system
data:
  fluentd.conf: |
    # 原有控制台日志采集配置（保留原有逻辑即可，不需要修改）
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
    </source>

    # 新增：自定义日志文件采集配置
    <source>
      @type tail
      path /var/log/app-custom-logs/*/*.log*
      pos_file /var/log/fluentd-custom-app.log.pos
      tag custom.app.*
      <parse>
        # 按你的Node.js日志格式调整，示例为通用text格式
        @type none
      </parse>
    </source>

    # 原有K8s元数据过滤配置（保留即可）
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    # 新增：自定义日志补全标识，方便在Kibana区分
    <filter custom.app.**>
      @type record_transformer
      <record>
        log_type "custom_app_log"
        service_name ${tag_parts[2]}
      </record>
    </filter>

    # 原有输出到Elasticsearch的配置（保留即可，会自动把两类日志都输出）
    <match **>
      @type elasticsearch
      host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
      port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
      logstash_format true
      logstash_prefix "kubernetes-log"
      <buffer>
        flush_mode interval
        flush_interval 5s
      </buffer>
    </match>

创建完ConfigMap后，需要在Fluentd DaemonSet的volumes和volumeMounts中新增ConfigMap的挂载，把配置文件挂载到Fluentd的配置目录/fluentd/etc/下即可。

验证方式 #

配置全部更新后，等待Pod重启完成，进入Kibana查看索引，可通过log_type:custom_app_log过滤自定义日志，原有控制台日志不受影响。

内容的提问来源于stack exchange，提问作者Sathish