如何将EKS上SpringBoot Pod的日志通过Filebeat发送至AWS Elasticsearch

下面是一套完整的落地步骤，涵盖部署、配置到验证的全流程：

1. 在EKS集群部署Filebeat（DaemonSet模式） #

Filebeat以DaemonSet方式部署，确保每个EKS节点都运行一个实例，用于采集节点上的容器日志。

1.1 准备RBAC权限配置 #

创建filebeat-rbac.yaml，赋予Filebeat读取K8s资源和节点日志目录的权限：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
- apiGroups: [""]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

执行部署：kubectl apply -f filebeat-rbac.yaml

1.2 部署Filebeat DaemonSet #

创建filebeat-daemonset.yaml，重点挂载日志存储目录和配置文件：

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.10.0 # 选择对应版本
        args: [
          "-c", "/usr/share/filebeat/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: filebeat-config
          mountPath: /usr/share/filebeat/filebeat.yml
          subPath: filebeat.yml
      volumes:
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: filebeat-config
        configMap:
          name: filebeat-config

执行部署：kubectl apply -f filebeat-daemonset.yaml

2. 配置Filebeat采集SpringBoot日志 #

创建ConfigMapfilebeat-config.yaml，指定采集规则和过滤逻辑：

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |-
    filebeat.modules:
    - module: kubernetes
      pod:
        enabled: true
        paths:
          - /var/log/pods/*/*/*.log
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
              - logs_path:
                  logs_path: "/var/log/pods/"
          # 只采集你的SpringBoot应用日志，替换成实际的label键值对
          - drop_event.when.not.equals:
              kubernetes.labels.app: "your-springboot-app"
          # 如果SpringBoot日志是JSON格式，解析成结构化数据
          - decode_json_fields:
              fields: ["message"]
              target: ""
              overwrite_keys: true

    setup.template.enabled: false
    setup.ilm.enabled: false

执行部署：kubectl apply -f filebeat-config.yaml

3. 配置Filebeat输出到AWS Elasticsearch #

AWS ES需要IAM权限认证，推荐使用IRSA（IAM Roles for Service Accounts）方式，避免硬编码密钥。

3.1 绑定IRSA权限 #

1. 创建IAM角色，赋予AmazonESFullAccess（生产环境建议更细粒度的权限，比如允许es:ESHttpPut、es:ESHttpPost等）
2. 通过eksctl或AWS CLI将该IAM角色关联到Filebeat的Service Account：

eksctl create iamserviceaccount \
  --cluster=your-eks-cluster-name \
  --namespace=kube-system \
  --name=filebeat \
  --attach-policy-arn=arn:aws:iam::your-account-id:policy/your-es-policy \
  --approve \
  --override-existing-serviceaccounts

3.2 更新Filebeat输出配置 #

修改filebeat-config.yaml的output部分：

output.elasticsearch:
  hosts: ["https://your-aws-es-endpoint"] # 替换为你的AWS ES域名
  protocol: "https"
  aws:
    enabled: true
    region: "your-aws-region" # 比如us-east-1

重新加载配置：kubectl apply -f filebeat-config.yaml

4. 验证日志传输 #

1. 检查Filebeat Pod运行状态：kubectl get pods -n kube-system -l k8s-app=filebeat
2. 查看Filebeat日志排查错误：kubectl logs -n kube-system filebeat-xxxxxx
3. 登录AWS ES对应的Kibana，创建索引模式（如filebeat-*），在Discover页面搜索SpringBoot应用的日志关键词，确认日志已正常传输。

内容的提问来源于stack exchange，提问作者zilcuanu