容器云平台No.9~kubernetes日志收集系统EFK

EFK介绍

EFK，全称Elasticsearch Fluentd Kibana ，是kubernetes中比较常用的日志收集方案，也是官方比较推荐的方案。

通过EFK，可以把集群的所有日志收集到Elasticsearch中，然后可以对日志做分析。一般用于故障排查，数据分析等。。。

数据流示意图

官方项目

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

小技巧，如果只希望下载github项目的某一个目录，可以使用svn，这里就只下载fluentd-elasticsearch目录，

例如需要下载的子目录为：

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

将/tree/master/换成trunk,然后使用svn下载即可

svn co https://github.com/kubernetes/kubernetes/trunk/cluster/addons/fluentd-elasticsearch

这里因为是学习，一步步安装，感兴趣的可以看官方项目

部署Elasticsearch

存储服务是基础，需要先部署，其他两个服务运行的时候需要连接es。

1、编写efk-es-statefulset.yaml

---

apiVersion: v1

kind: Namespace

metadata:

  name: efk

---

kind: Service

apiVersion: v1

metadata:

  name: elasticsearch-logging

  namespace: efk

  labels:

    app: elasticsearch-logging

spec:

  selector:

    app: elasticsearch-logging

  clusterIP: None

  ports:

    - port: 9200

      name: rest

    - port: 9300

      name: inter

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: elasticsearch-logging

  namespace: efk

spec:

  serviceName: elasticsearch-logging

  replicas: 3

  selector:

    matchLabels:

      app: elasticsearch-logging

  template:

    metadata:

      labels:

        app: elasticsearch-logging

    spec:

      initContainers:

      - name: increase-vm-max-map

        image: busybox

        command: ["sysctl", "-w", "vm.max_map_count=262144"]

        securityContext:

          privileged: true

      - name: increase-fd-ulimit

        image: busybox

        command: ["sh", "-c", "ulimit -n 65536"]

        securityContext:

          privileged: true

      containers:

      - name: elasticsearch-logging

        image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1

        ports:

        - name: rest

          containerPort: 9200

        - name: inter

          containerPort: 9300

        resources:

          limits:

            cpu: 1000m

          requests:

            cpu: 1000m

        volumeMounts:

        - name: elasticsearch-logging

          mountPath: /usr/share/elasticsearch/data

        env:

        - name: cluster.name

          value: k8s-logs

        - name: node.name

          valueFrom:

            fieldRef:

              fieldPath: metadata.name

        - name: cluster.initial_master_nodes

          value: "elasticsearch-logging-0,elasticsearch-logging-1,elasticsearch-logging-2"

        - name: discovery.zen.minimum_master_nodes

          value: "2"

        - name: discovery.seed_hosts

          value: "elasticsearch-logging"

        - name: ES_JAVA_OPTS

          value: "-Xms512m -Xmx512m"

        - name: network.host

          value: "0.0.0.0"

      volumes:

      - name: elasticsearch-logging

        emptyDir: {}

2、执行部署命令

这里需要注意，如果长时间下载不下来镜像，可以自行先将镜像下载，要不然可能会一直不成功

本文把服务都部署到命名空间:efk

[root@k8s-master001 EFK]# kubectl  apply -f efk-es-statefulset.yaml

[root@k8s-node001 EFK]# kubectl  get po -n efk

NAME                      READY   STATUS    RESTARTS   AGE

elasticsearch-logging-0   1/1     Running   0          10m

elasticsearch-logging-1   1/1     Running   0          10m

elasticsearch-logging-2   1/1     Running   0          9m42s

3、验证es是否正常运行

sh-4.2# curl http://localhost:9200/_cluster/state?pretty

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{

  "cluster_name" : "k8s-logs",

  "cluster_uuid" : "OLzzi6sbSZG11bqBFM9z5Q",

  "version" : 38,

  "state_uuid" : "uFa1_QKgRAK_NJ33SArGDw",

  "master_node" : "XiShXS0DSGmx0Dxp1r9vEw",

  "blocks" : { },

  "nodes" : {

    "XN-vHccLRkaEgr9Q1cctNA" : {

      "name" : "elasticsearch-logging-2",

      "ephemeral_id" : "WBEY2tGNRzmc3cBDJAEP9Q",

      "transport_address" : "100.108.163.2:9300",

      "attributes" : {

        "ml.machine_memory" : "16630661120",

        "ml.max_open_jobs" : "20",

        "xpack.installed" : "true",

        "transform.node" : "true"

      }

    },

.................

以上, elasticsearch就部署好了，接下来部署kibana

部署kibana

1、编写efk-kibana.yaml

apiVersion: v1

kind: Service

metadata:

  name: kibana-logging

  namespace: efk

  labels:

    app: kibana-logging

spec:

  ports:

  - port: 5601

  type: NodePort

  selector:

    app: kibana-logging

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: kibana-logging

  namespace: efk

  labels:

    app: kibana-logging

spec:

  selector:

    matchLabels:

      app: kibana-logging

  template:

    metadata:

      labels:

        app: kibana-logging

    spec:

      containers:

      - name: kibana-logging

        image: docker.elastic.co/kibana/kibana:7.9.1

        resources:

          limits:

            cpu: 1000m

          requests:

            cpu: 1000m

        env:

        - name: ELASTICSEARCH_HOSTS

          value: http://elasticsearch-logging:9200

        ports:

        - containerPort: 5601

2、执行部署命令

[root@k8s-node001 EFK]# kubectl  apply -f efk-kibana.yaml

service/kibana-logging created

deployment.apps/kibana-logging created

[root@k8s-node001 EFK]# kubectl  get po,svc -n efk

NAME                              READY   STATUS    RESTARTS   AGE

kibana-logging-6b5f984c44-7ljjn   1/1     Running   0          8m16s

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE

service/elasticsearch-logging   ClusterIP   None            <none>        9200/TCP,9300/TCP   28m

service/kibana-logging          NodePort    10.105.208.90   <none>        5601:32352/TCP      13m

3、验证kibana

服务以及通过NodePort暴露，通过IP+32352，可以访问到kibana web界面，如图所示

下一步，我们来部署日志收集客户端Fluentd

部署Fluentd

1、使用configmap创建fluentd配置文件

配置比较长，可以查看链接，这里就不贴出来了

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml

2、执行部署

[root@k8s-node001 EFK]# kubectl  appply -f fluentd-es-configmap.yaml

3、创建fluentd-es-ds.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: fluentd-es

  namespace: efk

  labels:

    k8s-app: fluentd-es

    addonmanager.kubernetes.io/mode: Reconcile

---

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: fluentd-es

  labels:

    k8s-app: fluentd-es

    addonmanager.kubernetes.io/mode: Reconcile

rules:

- apiGroups:

  - ""

  resources:

  - "namespaces"

  - "pods"

  verbs:

  - "get"

  - "watch"

  - "list"

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: fluentd-es

  labels:

    k8s-app: fluentd-es

    addonmanager.kubernetes.io/mode: Reconcile

subjects:

- kind: ServiceAccount

  name: fluentd-es

  namespace: efk

  apiGroup: ""

roleRef:

  kind: ClusterRole

  name: fluentd-es

  apiGroup: ""

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: fluentd-es-v3.0.2

  namespace: efk

  labels:

    k8s-app: fluentd-es

    version: v3.0.2

    addonmanager.kubernetes.io/mode: Reconcile

spec:

  selector:

    matchLabels:

      k8s-app: fluentd-es

      version: v3.0.2

  template:

    metadata:

      labels:

        k8s-app: fluentd-es

        version: v3.0.2

    spec:

      securityContext:

        seccompProfile:

          type: RuntimeDefault

      priorityClassName: system-node-critical

      serviceAccountName: fluentd-es

      containers:

      - name: fluentd-es

        image: registry.cn-qingdao.aliyuncs.com/up2cloud/fluentd:v3.0.2

        env:

        - name: FLUENTD_ARGS

          value: --no-supervisor -q

        resources:

          limits:

            memory: 500Mi

          requests:

            cpu: 100m

            memory: 200Mi

        volumeMounts:

        - name: varlog

          mountPath: /var/log

        - name: varlibdockercontainers

          mountPath: /var/lib/docker/containers

          readOnly: true

        - name: config-volume

          mountPath: /etc/fluent/config.d

        ports:

        - containerPort: 24231

          name: prometheus

          protocol: TCP

        livenessProbe:

          tcpSocket:

            port: prometheus

          initialDelaySeconds: 5

          timeoutSeconds: 10

        readinessProbe:

          tcpSocket:

            port: prometheus

          initialDelaySeconds: 5

          timeoutSeconds: 10

      terminationGracePeriodSeconds: 30

      volumes:

      - name: varlog

        hostPath:

          path: /var/log

      - name: varlibdockercontainers

        hostPath:

          path: /var/lib/docker/containers

      - name: config-volume

        configMap:

          name: fluentd-es-config-v0.2.0

4、执行部署

[root@k8s-node001 EFK]# kubectl apply -f fluentd-es-ds.yaml

5、查看部署结果

[root@k8s-node001 EFK]# kubectl  get po -n efk

NAME                              READY   STATUS    RESTARTS   AGE

elasticsearch-logging-0           1/1     Running   0          3h34m

elasticsearch-logging-1           1/1     Running   0          3h33m

elasticsearch-logging-2           1/1     Running   0          3h33m

fluentd-es-v3.0.2-24lbr           1/1     Running   0          26m

fluentd-es-v3.0.2-5qcsv           1/1     Running   0          26m

fluentd-es-v3.0.2-gnp58           1/1     Running   0          26m

fluentd-es-v3.0.2-gtx4s           1/1     Running   0          26m

fluentd-es-v3.0.2-mxz9t           1/1     Running   0          26m

kibana-logging-6b5f984c44-7ljjn   1/1     Running   0          3h19m

从输出信息可以看到，整套日志收集系统已经全部正常运行，现在就可以使用kibana查看收集到的日志了

至此日志收集系统搭建完毕，EFK更多用途后面会陆续介绍，也可以自行前往官网查看。

巴特西