上一篇文章

IoT 边缘集群基于 Kubernetes Events 的告警通知实现

目标

1. 告警恢复通知 - 经过评估无法实现
   1. 原因: 告警和恢复是单独完全不相关的事件, 告警是 Warning 级别, 恢复是 Normal 级别, 要开启恢复, 就会导致所有 Normal Events 都会被发送, 这个数量是很恐怖的; 而且, 除非特别有经验和耐心, 否则无法看出哪条 Normal 对应的是 告警的恢复.
2. 未恢复进行持续告警 - 默认就带的能力, 无需额外配置.
3. 告警内容显示资源名称，比如节点和pod名称
4. 可以设置屏蔽特定的节点和工作负载并可以动态调整
   1. 比如，集群001中的节点worker-1做计划性维护，期间停止监控，维护完成后重新开始监控。

配置

告警内容显示资源名称

典型的几类 events:

apiVersion: v1

count: 101557

eventTime: null

firstTimestamp: "2022-04-08T03:50:47Z"

involvedObject:

  apiVersion: v1

  fieldPath: spec.containers{prometheus}

  kind: Pod

  name: prometheus-rancher-monitoring-prometheus-0

  namespace: cattle-monitoring-system

kind: Event

lastTimestamp: "2022-04-14T11:39:19Z"

message: 'Readiness probe failed: Get "http://10.42.0.87:9090/-/ready": context deadline

  exceeded (Client.Timeout exceeded while awaiting headers)'

metadata:

  creationTimestamp: "2022-04-08T03:51:17Z"

  name: prometheus-rancher-monitoring-prometheus-0.16e3cf53f0793344

  namespace: cattle-monitoring-system

reason: Unhealthy

reportingComponent: ""

reportingInstance: ""

source:

  component: kubelet

  host: master-1

type: Warning

apiVersion: v1

count: 116

eventTime: null

firstTimestamp: "2022-04-13T02:43:26Z"

involvedObject:

  apiVersion: v1

  fieldPath: spec.containers{grafana}

  kind: Pod

  name: rancher-monitoring-grafana-57777cc795-2b2x5

  namespace: cattle-monitoring-system

kind: Event

lastTimestamp: "2022-04-14T11:18:56Z"

message: 'Readiness probe failed: Get "http://10.42.0.90:3000/api/health": context

  deadline exceeded (Client.Timeout exceeded while awaiting headers)'

metadata:

  creationTimestamp: "2022-04-14T11:18:57Z"

  name: rancher-monitoring-grafana-57777cc795-2b2x5.16e5548dd2523a13

  namespace: cattle-monitoring-system

reason: Unhealthy

reportingComponent: ""

reportingInstance: ""

source:

  component: kubelet

  host: master-1

type: Warning

apiVersion: v1

count: 20958

eventTime: null

firstTimestamp: "2022-04-11T10:34:51Z"

involvedObject:

  apiVersion: v1

  fieldPath: spec.containers{lb-port-1883}

  kind: Pod

  name: svclb-emqx-dt22t

  namespace: emqx

kind: Event

lastTimestamp: "2022-04-14T11:39:48Z"

message: Back-off restarting failed container

metadata:

  creationTimestamp: "2022-04-11T10:34:51Z"

  name: svclb-emqx-dt22t.16e4d11e2b9efd27

  namespace: emqx

reason: BackOff

reportingComponent: ""

reportingInstance: ""

source:

  component: kubelet

  host: worker-1

type: Warning

apiVersion: v1

count: 21069

eventTime: null

firstTimestamp: "2022-04-11T10:34:48Z"

involvedObject:

  apiVersion: v1

  fieldPath: spec.containers{lb-port-80}

  kind: Pod

  name: svclb-traefik-r5p8t

  namespace: kube-system

kind: Event

lastTimestamp: "2022-04-14T11:44:59Z"

message: Back-off restarting failed container

metadata:

  creationTimestamp: "2022-04-11T10:34:48Z"

  name: svclb-traefik-r5p8t.16e4d11daf0b79ce

  namespace: kube-system

reason: BackOff

reportingComponent: ""

reportingInstance: ""

source:

  component: kubelet

  host: worker-1

type: Warning

{

  "metadata": {

    "name": "event-exporter-79544df9f7-xj4t5.16e5c540dc32614f",

    "namespace": "monitoring",

    "uid": "baf2f642-2383-4e22-87e0-456b6c3eaf4e",

    "resourceVersion": "14043444",

    "creationTimestamp": "2022-04-14T13:08:40Z"

  },

  "reason": "Pulled",

  "message": "Container image \"ghcr.io/opsgenie/kubernetes-event-exporter:v0.11\" already present on machine",

  "source": {

    "component": "kubelet",

    "host": "worker-2"

  },

  "firstTimestamp": "2022-04-14T13:08:40Z",

  "lastTimestamp": "2022-04-14T13:08:40Z",

  "count": 1,

  "type": "Normal",

  "eventTime": null,

  "reportingComponent": "",

  "reportingInstance": "",

  "involvedObject": {

    "kind": "Pod",

    "namespace": "monitoring",

    "name": "event-exporter-79544df9f7-xj4t5",

    "uid": "b77d3e13-fa9e-484b-8a5a-d1afc9edec75",

    "apiVersion": "v1",

    "resourceVersion": "14043435",

    "fieldPath": "spec.containers{event-exporter}",

    "labels": {

      "app": "event-exporter",

      "pod-template-hash": "79544df9f7",

      "version": "v1"

    }

  }

}

我们可以把更多的字段加入到告警信息中, 其中就包括:

• 节点: {{ Source.Host }}
• Pod: {{ .InvolvedObject.Name }}

综上, 修改后的event-exporter-cfg yaml 如下:

apiVersion: v1

kind: ConfigMap

metadata:

  name: event-exporter-cfg

  namespace: monitoring

  resourceVersion: '5779968'

data:

  config.yaml: |

    logLevel: error

    logFormat: json

    route:

      routes:

        - match:

            - receiver: "dump"

        - drop:

            - type: "Normal"

          match:

            - receiver: "feishu"

    receivers:

      - name: "dump"

        stdout: {}

      - name: "feishu"

        webhook:

          endpoint: "https://open.feishu.cn/open-apis/bot/v2/hook/..."

          headers:

            Content-Type: application/json

          layout:

            msg_type: interactive

            card:

              config:

                wide_screen_mode: true

                enable_forward: true

              header:

                title:

                  tag: plain_text

                  content: xxx测试K3S集群告警

                template: red

              elements:

                - tag: div

                  text:

                    tag: lark_md

                    content: "**EventID:**  {{ .UID }}\n**EventNamespace:**  {{ .InvolvedObject.Namespace }}\n**EventName:**  {{ .InvolvedObject.Name }}\n**EventType:**  {{ .Type }}\n**EventKind:**  {{ .InvolvedObject.Kind }}\n**EventReason:**  {{ .Reason }}\n**EventTime:**  {{ .LastTimestamp }}\n**EventMessage:**  {{ .Message }}\n**EventComponent:**  {{ .Source.Component }}\n**EventHost:**  {{ .Source.Host }}\n**EventLabels:**  {{ toJson .InvolvedObject.Labels}}\n**EventAnnotations:**  {{ toJson .InvolvedObject.Annotations}}"

屏蔽特定的节点和工作负载

比如，集群001中的节点worker-1做计划性维护，期间停止监控，维护完成后重新开始监控。

继续修改event-exporter-cfg yaml 如下:

apiVersion: v1

kind: ConfigMap

metadata:

  name: event-exporter-cfg

  namespace: monitoring

data:

  config.yaml: |

    logLevel: error

    logFormat: json

    route:

      routes:

        - match:

            - receiver: "dump"

        - drop:

            - type: "Normal"

            - source:

                host: "worker-1"

            - namespace: "cattle-monitoring-system"

            - name: "*emqx*"

            - kind: "Pod|Deployment|ReplicaSet"

            - labels:

                version: "dev"

          match:

            - receiver: "feishu"

    receivers:

      - name: "dump"

        stdout: {}

      - name: "feishu"

        webhook:

          endpoint: "https://open.feishu.cn/open-apis/bot/v2/hook/..."

          headers:

            Content-Type: application/json

          layout:

            msg_type: interactive

            card:

              config:

                wide_screen_mode: true

                enable_forward: true

              header:

                title:

                  tag: plain_text

                  content: xxx测试K3S集群告警

                template: red

              elements:

                - tag: div

                  text:

                    tag: lark_md

                    content: "**EventID:**  {{ .UID }}\n**EventNamespace:**  {{ .InvolvedObject.Namespace }}\n**EventName:**  {{ .InvolvedObject.Name }}\n**EventType:**  {{ .Type }}\n**EventKind:**  {{ .InvolvedObject.Kind }}\n**EventReason:**  {{ .Reason }}\n**EventTime:**  {{ .LastTimestamp }}\n**EventMessage:**  {{ .Message }}\n**EventComponent:**  {{ .Source.Component }}\n**EventHost:**  {{ .Source.Host }}\n**EventLabels:**  {{ toJson .InvolvedObject.Labels}}\n**EventAnnotations:**  {{ toJson .InvolvedObject.Annotations}}"

默认的 drop 规则为: - type: "Normal", 即不对 Normal 级别进行告警;

现在加入以下规则:

            - source:

                host: "worker-1"

            - namespace: "cattle-monitoring-system"

            - name: "*emqx*"

            - kind: "Pod|Deployment|ReplicaSet"

            - labels:

                version: "dev"

• ... host: "worker-1": 不对节点worker-1 做告警;
• ... namespace: "cattle-monitoring-system": 不对 NameSpace: cattle-monitoring-system 做告警;
• ... name: "*emqx*": 不对 name(name 往往是 pod name) 包含 emqx 的做告警
• kind: "Pod|Deployment|ReplicaSet": 不对 Pod Deployment ReplicaSet 做告警(也就是不关注应用, 组件相关的告警)
• ...version: "dev": 不对 label 含有 version: "dev" 的做告警(可以通过它屏蔽特定的应用的告警)

最终效果

如下图:

三人行, 必有我师; 知识共享, 天下为公. 本文由东风微鸣技术博客 EWhisper.cn 编写.

最新文章

1. RStudio技巧01_美化RStudio的帮助页面
2. 动态加载框架DL分析
3. Java构建工具Ant小记(一)
4. Unity本身制作keystore
5. python与正则表达式：re模块详解
6. C++主要数据类型在计算机中所占字节大小
7. Enterprise Solution 2.2 开发帮助文档集合
8. Linux14.04安装JDK
9. cookies的理解
10. ZooKeeper伪分布集群安装及使用 RMI+ZooKeeper实现远程调用框架
11. （转）Const，Const函数，Const变量，函数后面的Const
12. Socket编程(九)
13. 微信小程序购物车产品计价
14. 面向对象_06【抽象类：abstract、接口：interface、实现：implements】
15. Effective Java 之-----返回零长度的数组或集合而不是null
16. Android热插拔事件处理详解
17. 手把手教Electron+vue的使用
18. node.js入门基础
19. 操作系统之Linux的内存分页管理
20. 开启session

热门文章

1. 基于python的数学建模---传染病六个模型
2. C温故补缺(三):存储类声明符(auto,register,extern,static)
3. 【机器学习】李宏毅——卷积神经网络CNN
4. flutter系列之:移动端手势的具体使用
5. PowerDotNet平台化软件架构设计与实现系列（15）：支付平台
6. 1、mybatis逆向工程
7. 万字长文解析Scaled YOLOv4模型(YOLO变体模型)
8. Java关键词synchronized解读
9. JUC源码学习笔记7——FutureTask源码解析,人生亦如是，run起来才有结果
10. NOIP2022 退役记