prometheus Alertmanager webhook

一.自定义邮件告警

在alertmanager服务的配置文件中指定自定义告警文件

# vim alertmanager.yaml

global:

   resolve_timeout: 5m

   smtp_smarthost: 'smtp.qq.com:465'

   smtp_from: 'x34989xxx@qq.com'

   smtp_auth_username: 'x34989xx@qq.com'

   smtp_auth_password: 'lxxxxxxxxtubdfd'

   smtp_require_tls: false

   smtp_hello: 'qq.com'

templates:

   - '/etc/alertmanager/template/*.tmpl'

route:

   group_by: ['alertname']

   group_wait: 10s

   group_interval: 10s

   repeat_interval: 1m

   receiver: 'email'

receivers:

   - name: 'email'

     email_configs:

     - to: '20xxxxxx@qq.com'

       html: '{{ template "email.to.html" . }}'

       headers: { Subject: "告警" }

       send_resolved: true    

inhibit_rules:

   - source_match:

       severity: 'critical'

     target_match:

       severity: 'warning'

     equal: ['alertname', 'dev', 'instance']

书写自定义配置文件

{{ define "email.to.html" }}

{{ if gt (len .Alerts.Firing) 0 }}

{{ range .Alerts }}

========  异常告警========== <br>

告警程序: Alertmanager <br>

告警主机: {{ .Annotations.summary }} <br>

告警类型: {{ .Annotations.alarmPolicyType }} <br>

告警级别: {{ .Labels.severity }} <br>

告警状态: {{ .Status }} <br>

告警详情: {{ .Annotations.description }} <br>

触发时间: {{ (.StartsAt.Add 28800e9) "2022-8-11 17:30:01" }} <br>

==========end============= <br>

{{ end }}

{{ end }}

{{ if gt (len .Alerts.Resolved) 0 }}

{{ range .Alerts }}

======== <span style=color:#00FF00;font-size:11px;font-weight:bold;> 告警恢复 </span>==========<br>

告警程序: Alertmanager <br>

告警主机: {{ .Annotations.summary }} <br>

告警类型: {{ .Annotations.alarmPolicyType }} <br>

告警级别: {{ .Labels.severity }} <br>

告警状态: {{ .Status }} <br>

告警详情: {{ .Annotations.description }} <br>

触发时间: {{ (.StartsAt.Add 28800e9) "2022-8-11 17:30:01" }} <br>

恢复时间: {{ (.EndsAt.Add 28800e9) "2022-8-11 17:30:01" }} <br>

===========end============ <br>

{{ end }}

{{ end }}

{{ end }}

注：里面的标签需要根据内置的标签和自定义的标签来配置

二.使用docker部署微信机器人告警

注：这里的自定义告警用webhook来转发

1.制作镜像

1)准备webhook启动代码

注：里面的标签需要和自己的配置的相同

~]# vim app.py

# -*- coding: utf-8 -*-

import os

import json

import requests

import arrow

from flask import Flask

from flask import request

app = Flask(__name__)

def bytes2json(data_bytes):

    data = data_bytes.decode('utf8').replace("'", '"')

    return json.loads(data)

def makealertdata(data):

    for output in data['alerts'][:]:

        try:

            pod_name = output['labels']['pod']

        except KeyError:

            try:

                pod_name = output['labels']['pod_name']

            except KeyError:

                pod_name = 'null'

        try:

            namespace = output['labels']['namespace']

        except KeyError:

            namespace = 'null'

        try:

            message = output['annotations']['message']

        except KeyError:

            try:

                message = output['annotations']['description']

            except KeyError:

                message = 'null'

        if output['status'] == 'firing':

            status_zh = '<font color=\"warning\">告警</font>'

            title = '【%s】告警 %s 有新的报警' % (status_zh, output['annotations']['alarmPolicyType'])

            send_data = {

                "msgtype": "markdown",

                "markdown": {

                    "content": "## %s \n\n" %title +

                            ">**告警级别**: %s \n\n" % output['labels']['severity'] +

                            ">**告警类型**: %s \n\n" % output['annotations']['metricDisplayName'] +

                            ">**告警主机**: %s \n\n" % output['annotations']['summary'] +

                            ">**告警负责人**: %s \n\n" % "<@v_keo>" +

                            ">**告警详情**: %s \n\n" % message +

                            ">**告警状态**: %s \n\n" % output['status'] +

                            ">**触发时间**: %s \n\n" % arrow.get(output['startsAt']).to('Asia/Shanghai').format(

                        'YYYY-MM-DD HH:mm:ss ZZ')

                }

            }

        elif output['status'] == 'resolved':

            status_zh = '<font color=\"info\">恢复</font>'

            title = '【%s】环境 %s 有报警恢复' % (status_zh, output['annotations']['alarmPolicyType'])

            send_data = {

                "msgtype": "markdown",

                "markdown": {

                    "content": "## %s \n\n" %title +

                            ">**告警级别**: %s \n\n" % output['labels']['severity'] +

                            ">**告警类型**: %s \n\n" % output['annotations']['metricDisplayName'] +

                            ">**告警主机**: %s \n\n" % output['annotations']['summary'] +

                            ">**告警负责人**: %s \n\n" % "<@v_keo>" +

                            ">**告警详情**: %s \n\n" % message +

                            ">**告警状态**: %s \n\n" % output['status'] +

                            ">**触发时间**: %s \n\n" % arrow.get(output['startsAt']).to('Asia/Shanghai').format(

                        'YYYY-MM-DD HH:mm:ss ZZ') +

                            ">**触发结束时间**: %s \n" % arrow.get(output['endsAt']).to('Asia/Shanghai').format(

                        'YYYY-MM-DD HH:mm:ss ZZ')

                }

            }

        return send_data

def send_alert(data):

    token = os.getenv('ROBOT_TOKEN')

    if not token:

        print('you must set ROBOT_TOKEN env')

        return

    url = 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=%s' % token

    send_data = makealertdata(data)

    req = requests.post(url, json=send_data)

    result = req.json()

    if result['errcode'] != 0:

        print('notify dingtalk error: %s' % result['errcode'])

@app.route('/', methods=['POST', 'GET'])

def send():

    if request.method == 'POST':

        post_data = request.get_data()

        send_alert(bytes2json(post_data))

        return 'success'

    else:

        return 'weclome to use prometheus alertmanager dingtalk webhook server!'

if __name__ == '__main__':

    app.run(host='0.0.0.0', port=5000)

2）准备python需要的模块文件

~]# vim requirements.txt

certifi==2018.10.15

chardet==3.0.4

Click==7.0

Flask==1.0.2

idna==2.7

itsdangerous==1.1.0

Jinja2==2.10

MarkupSafe==1.1.0

requests==2.20.1

urllib3==1.24.1

Werkzeug==0.14.1

arrow==0.13.1

3）书写dockerfile文件

~]# vim Dockerfile

FROM python:3.6.4

# set working directory

WORKDIR /src

# add app

ADD . /src

# install requirements

RUN pip install selectivesearch -i http://pypi.douban.com/simple --trusted-host pypi.douban.com

RUN pip install -r requirements.txt

EXPOSE 500

# run server

CMD python app.py

4)制作镜像

docker build -f /data/prometheus/Dockerfile2/webkook/Dockerfile -t webhook:v1 . --network=host

# -f ： 指定dockerfile文件位置

2.启动容器和指定webhook容器

1）启动容器

docker run --name=webhook --net=host -v /etc/localtime:/etc/localtime -v /data/prometheus/Dockerfile2/webkook/app.py:/src/app.py -d -e ROBOT_TOKEN=xxxxx4-69ac-458c-af07-a69xxebsxxd webhook:v1

# /etc/localtime 时间要一致

# -e ROBOT_TOKEN= 指定机器人 key

2）告警配置指定webhook

]# cat alertmanager.yaml-wx 

global:

       resolve_timeout: 5m

route:

      group_by: ['alertname']

      group_wait: 10s

      group_interval: 10s

      repeat_interval: 1m

      receiver: 'web.hook'

receivers:

    - name: 'web.hook'

      webhook_configs:

      - url: 'http://起容器的ip:5000'

        send_resolved: true

inhibit_rules:

      - source_match:

           severity: 'critical'

        target_match:

           severity: 'warning'

        equal: ['alertname', 'dev', 'instance']

注：如果转发不了记得把端口映射出来

巴特西

prometheus Alertmanager webhook

一.自定义邮件告警

二.使用docker部署微信机器人告警

1.制作镜像

2.启动容器和指定webhook容器

最新文章

热门文章