Filebeat 收集K8S 日志，生产环境实践

根据生产环境要求，需要采集K8Spod 日志，和开发协商之后，pod中应用会将日志输出到容器终端上，这时可以直接用filebeat 采集node节点上面的/var/log/containers/*.log日志，然后将日志输出到kafka消息队列中，经过kafka将日志写入logstash进行格式化，然后由logstash传入elasticsearch存储，然后kibana会连接elasticsearch展示索引数据。

数据传输流程：Pod -> /var/log/containers/*.log -> Filebeat -> Kafka集群 -> Logstash -> Elasticsearch -> Kibana

K8S 配置Filebeat

整体配置文件如下：

$ ls

filebeat.daemonset.yml                   filebeat.permission.yml

filebeat.indice-lifecycle.configmap.yml  filebeat.settings.configmap.yml

Filebeat操作权限

$ cat filebeat.permission.yml

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

  name: filebeat

subjects:

- kind: ServiceAccount

  name: filebeat

  namespace: kube-system

roleRef:

  kind: ClusterRole

  name: filebeat

  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRole

metadata:

  name: filebeat

  labels:

    app: filebeat

rules:

- apiGroups: [""]

  resources:

  - namespaces

  - pods

  verbs:

  - get

  - watch

  - list

---

apiVersion: v1

kind: ServiceAccount

metadata:

  namespace: kube-system

  name: filebeat

  labels:

    app: filebeat

Filebeat主配置文件

注意：如果收集Java堆栈错误日志，需要增加下面带注释的几行参数，multiline多行处理解决次问题。

$ cat filebeat.settings.configmap.yml

---

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: kube-system

  name: filebeat-config

  labels:

    app: filebeat

data:

  filebeat.yml: |-

    filebeat.inputs:

    - type: container

      enabled: true

      paths:

      - /var/log/containers/*.log

      multiline: # 多行处理，正则表示如果前面几个数字不是4个数字开头，那么就会合并到一行,解决Java堆栈错误日志收集问题

        pattern: ^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2} #匹配Java日志开头时间

        negate: true # 正则是否开启，默认false不开启

        match: after # 不匹配的正则的行是放在上面一行的前面还是后面

      processors:

      - add_kubernetes_metadata:

          in_cluster: true

          host: ${NODE_NAME}

          matchers:

          - logs_path:

              logs_path: "/var/log/containers/"

      - add_cloud_metadata:

      - add_kubernetes_metadata:

          matchers:

          - logs_path:

              logs_path: "/var/log/containers/"

      - add_docker_metadata:

    output:

      kafka:

        enabled: true # 增加kafka的输出

        hosts: ["10.0.0.72:9092"]

        topic: filebeat

        max_message_bytes: 5242880

        partition.round_robin:

          reachable_only: true

        keep-alive: 120

        required_acks: 1

    setup.ilm:

      policy_file: /etc/indice-lifecycle.json

Filebeat索引生命周期策略配置

ElasticSearch 的 indice 生命周期表示一组规则，可以根据 indice 的大小或者时长应用到你的 indice 上。比如可以每天或者每次超过 1GB 大小的时候对 indice 进行轮转，我们也可以根据规则配置不同的阶段。由于监控会产生大量的数据，很有可能一天就超过几十G的数据，所以为了防止大量的数据存储，我们可以利用 indice 的生命周期来配置数据保留，这个在 Prometheus 中也有类似的操作。如下所示的文件中，我们配置成每天或每次超过5GB的时候就对 indice 进行轮转，并删除所有超过30天的 indice 文件，我们这里只保留30天监控数据完全足够了。

filebeat.indice-lifecycle.configmap.yml

---

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: kube-system

  name: filebeat-indice-lifecycle

  labels:

    app: filebeat

data:

  indice-lifecycle.json: |-

    {

      "policy": {

        "phases": {

          "hot": {

            "actions": {

              "rollover": {

                "max_size": "5GB" ,

                "max_age": "1d"

              }

            }

          },

          "delete": {

            "min_age": "30d",

            "actions": {

              "delete": {}

            }

          }

        }

      }

    }

Filebeat Daemonset配置文件

$ cat filebeat.daemonset.yml

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

  namespace: kube-system

  name: filebeat

  labels:

    app: filebeat

spec:

  selector:

    matchLabels:

      app: filebeat

  template:

    metadata:

      labels:

        app: filebeat

    spec:

      serviceAccountName: filebeat

      terminationGracePeriodSeconds: 30

      containers:

      - name: filebeat

        image: docker.elastic.co/beats/filebeat:7.8.0

        args: [

          "-c", "/etc/filebeat.yml",

          "-e",

        ]

        env:

        - name: NODE_NAME

          valueFrom:

            fieldRef:

              fieldPath: spec.nodeName

        securityContext:

          runAsUser: 0

        resources:

          limits:

            memory: 200Mi

          requests:

            cpu: 100m

            memory: 100Mi

        volumeMounts:

        - name: config

          mountPath: /etc/filebeat.yml

          readOnly: true

          subPath: filebeat.yml

        - name: filebeat-indice-lifecycle

          mountPath: /etc/indice-lifecycle.json

          readOnly: true

          subPath: indice-lifecycle.json

        - name: data

          mountPath: /usr/share/filebeat/data

        - name: varlog

          mountPath: /var/log

          readOnly: true

        - name: varlibdockercontainers

          mountPath: /var/lib/docker/containers

          readOnly: true

        - name: dockersock

          mountPath: /var/run/docker.sock

      volumes:

      - name: config

        configMap:

          defaultMode: 0600

          name: filebeat-config

      - name: filebeat-indice-lifecycle

        configMap:

          defaultMode: 0600

          name: filebeat-indice-lifecycle

      - name: varlog

        hostPath:

          path: /var/log

      - name: varlibdockercontainers

        hostPath:

          path: /var/lib/docker/containers

      - name: dockersock

        hostPath:

          path: /var/run/docker.sock

      - name: data

        hostPath:

          path: /var/lib/filebeat-data

          type: DirectoryOrCreate

执行到K8S中

$ kubectl apply  -f filebeat.settings.configmap.yml \

                 -f filebeat.indice-lifecycle.configmap.yml \

                 -f filebeat.daemonset.yml \

                 -f filebeat.permissions.yml 

configmap/filebeat-config created

configmap/filebeat-indice-lifecycle created

daemonset.apps/filebeat created

clusterrolebinding.rbac.authorization.k8s.io/filebeat created

clusterrole.rbac.authorization.k8s.io/filebeat created

serviceaccount/filebeat created

巴特西