• prometheus自定义邮件告警和自定义微信机器人告警

    目录

    ————————————————————————————

    一.自定义邮件告警

    二.使用docker部署微信机器人告警

    1.制作镜像

    2.启动容器和指定webhook容器

    ————————————————————————————

    一.自定义邮件告警

    在alertmanager服务的配置文件中指定自定义告警文件

    1. ~]# vim alertmanager.yaml
    2. global:
    3.    resolve_timeout: 5m
    4.    smtp_smarthost: 'smtp.qq.com:465'                                                      
    5.    smtp_from: 'x34989xxx@qq.com'                                                          
    6.    smtp_auth_username: 'x34989xx@qq.com'                                                 
    7.    smtp_auth_password: 'lxxxxxxxxtubdfd'                                                 
    8.    smtp_require_tls: false                                                                
    9.    smtp_hello: 'qq.com'
    10.  
    11. templates:
    12.    - '/etc/alertmanager/template/*.tmpl'
    13.  
    14. route:
    15.    group_by: ['alertname']
    16.    group_wait: 10s
    17.    group_interval: 10s
    18.    repeat_interval: 1m
    19.    receiver: 'email'
    20.  
    21. receivers:
    22.    - name: 'email'
    23.      email_configs:                                                                          
    24.      - to: '20xxxxxx@qq.com'                                                               
    25.        html: '{{ template "email.to.html" . }}'
    26.        headers: { Subject: "告警" }
    27.        send_resolved: true    
    28.  
    29. inhibit_rules:
    30.    - source_match:
    31.        severity: 'critical'
    32.      target_match:
    33.        severity: 'warning'
    34.      equal: ['alertname', 'dev', 'instance']

    书写自定义配置文件

    1. {{ define "email.to.html" }}
    2. {{ if gt (len .Alerts.Firing) 0 }}
    3. {{ range .Alerts }}
    4. ========  异常告警========== <br>
    5. 告警程序: Alertmanager <br>
    6. 告警主机: {{ .Annotations.summary }} <br>
    7. 告警类型: {{ .Annotations.alarmPolicyType }} <br>
    8. 告警级别: {{ .Labels.severity }} <br>
    9. 告警状态: {{ .Status }} <br>
    10. 告警详情: {{ .Annotations.description }} <br>
    11. 触发时间: {{ (.StartsAt.Add 28800e9) "2022-8-11 17:30:01" }} <br>
    12. ==========end============= <br>
    13. {{ end }}
    14. {{ end }}
    15. {{ if gt (len .Alerts.Resolved) 0 }}
    16. {{ range .Alerts }}
    17. ======== <span style=color:#00FF00;font-size:11px;font-weight:bold;> 告警恢复 </span>==========<br>
    18. 告警程序: Alertmanager <br>
    19. 告警主机: {{ .Annotations.summary }} <br>
    20. 告警类型: {{ .Annotations.alarmPolicyType }} <br>
    21. 告警级别: {{ .Labels.severity }} <br>
    22. 告警状态: {{ .Status }} <br>
    23. 告警详情: {{ .Annotations.description }} <br>
    24. 触发时间: {{ (.StartsAt.Add 28800e9) "2022-8-11 17:30:01" }} <br>
    25. 恢复时间: {{ (.EndsAt.Add 28800e9) "2022-8-11 17:30:01" }} <br>
    26. ===========end============ <br>
    27. {{ end }}
    28. {{ end }}
    29. {{ end }}

     注：里面的标签需要根据内置的标签和自定义的标签来配置

    二.使用docker部署微信机器人告警

    注：这里的自定义告警用webhook来转发

    1.制作镜像

    1)准备webhook启动代码

    注：里面的标签需要和自己的配置的相同

    1. ~]# vim app.py 
    2. # -*- coding: utf-8 -*-
    3. import os
    4. import json
    5. import requests
    6. import arrow
    7.  
    8. from flask import Flask
    9. from flask import request
    10.  
    11. app = Flask(__name__)
    12.  
    13. def bytes2json(data_bytes):
    14.     data = data_bytes.decode('utf8').replace("'", '"')
    15.     return json.loads(data)
    16.  
    17. def makealertdata(data):
    18.     for output in data['alerts'][:]:
    19.         try:
    20.             pod_name = output['labels']['pod']
    21.         except KeyError:
    22.             try:
    23.                 pod_name = output['labels']['pod_name']
    24.             except KeyError:
    25.                 pod_name = 'null'
    26.  
    27.         try:
    28.             namespace = output['labels']['namespace']
    29.         except KeyError:
    30.             namespace = 'null'
    31.  
    32.         try:
    33.             message = output['annotations']['message']
    34.         except KeyError:
    35.             try:
    36.                 message = output['annotations']['description']
    37.             except KeyError:
    38.                 message = 'null'
    39.         if output['status'] == 'firing':
    40.             status_zh = '告警'
    41.             title = '【%s】告警 %s 有新的报警' % (status_zh, output['annotations']['alarmPolicyType'])
    42.             send_data = {
    43.                 "msgtype": "markdown",
    44.                 "markdown": {
    45.                     "content": "## %s \n\n" %title +
    46.                             ">**告警级别**: %s \n\n" % output['labels']['severity'] +
    47.                             ">**告警类型**: %s \n\n" % output['annotations']['metricDisplayName'] +
    48.                             ">**告警主机**: %s \n\n" % output['annotations']['summary'] +
    49.                             ">**告警负责人**: %s \n\n" % "<@v_keo>" +
    50.                             ">**告警详情**: %s \n\n" % message +
    51.                             ">**告警状态**: %s \n\n" % output['status'] +
    52.                             ">**触发时间**: %s \n\n" % arrow.get(output['startsAt']).to('Asia/Shanghai').format(
    53.                         'YYYY-MM-DD HH:mm:ss ZZ')
    54.                 }
    55.             }
    56.         elif output['status'] == 'resolved':
    57.             status_zh = '恢复'
    58.             title = '【%s】环境 %s 有报警恢复' % (status_zh, output['annotations']['alarmPolicyType'])
    59.             send_data = {
    60.                 "msgtype": "markdown",
    61.                 "markdown": {
    62.                     "content": "## %s \n\n" %title +
    63.                             ">**告警级别**: %s \n\n" % output['labels']['severity'] +
    64.                             ">**告警类型**: %s \n\n" % output['annotations']['metricDisplayName'] +
    65.                             ">**告警主机**: %s \n\n" % output['annotations']['summary'] +
    66.                             ">**告警负责人**: %s \n\n" % "<@v_keo>" +
    67.                             ">**告警详情**: %s \n\n" % message +
    68.                             ">**告警状态**: %s \n\n" % output['status'] +
    69.                             ">**触发时间**: %s \n\n" % arrow.get(output['startsAt']).to('Asia/Shanghai').format(
    70.                         'YYYY-MM-DD HH:mm:ss ZZ') +
    71.                             ">**触发结束时间**: %s \n" % arrow.get(output['endsAt']).to('Asia/Shanghai').format(
    72.                         'YYYY-MM-DD HH:mm:ss ZZ')
    73.                 }
    74.             }
    75.         return send_data
    76. def send_alert(data):
    77.     token = os.getenv('ROBOT_TOKEN')
    78.     if not token:
    79.         print('you must set ROBOT_TOKEN env')
    80.         return
    81.     url = 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=%s' % token
    82.  
    83.     send_data = makealertdata(data)
    84.     req = requests.post(url, json=send_data)
    85.     result = req.json()
    86.     if result['errcode'] != 0:
    87.         print('notify dingtalk error: %s' % result['errcode'])
    88.  
    89.  
    90. @app.route('/', methods=['POST', 'GET'])
    91. def send():
    92.     if request.method == 'POST':
    93.         post_data = request.get_data()
    94.         send_alert(bytes2json(post_data))
    95.         return 'success'
    96.     else:
    97.         return 'weclome to use prometheus alertmanager dingtalk webhook server!'
    98.  
    99.  
    100. if __name__ == '__main__':
    101.     app.run(host='0.0.0.0', port=5000)

    2）准备python需要的模块文件

    1. ~]# vim requirements.txt 
    2. certifi==2018.10.15
    3. chardet==3.0.4
    4. Click==7.0
    5. Flask==1.0.2
    6. idna==2.7
    7. itsdangerous==1.1.0
    8. Jinja2==2.10
    9. MarkupSafe==1.1.0
    10. requests==2.20.1
    11. urllib3==1.24.1
    12. Werkzeug==0.14.1
    13. arrow==0.13.1

    3）书写dockerfile文件

    1. ~]# vim Dockerfile 
    2. FROM python:3.6.4
    3. # set working directory
    4. WORKDIR /src
    5. # add app
    6. ADD . /src
    7. # install requirements
    8. RUN pip install selectivesearch -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
    9. RUN pip install -r requirements.txt
    10. EXPOSE 500
    11. # run server
    12. CMD python app.py

    4)制作镜像

    1. ~]# docker build -f /data/prometheus/Dockerfile2/webkook/Dockerfile -t  webhook:v1 . --network=host
    2. # -f ： 指定dockerfile文件位置

    2.启动容器和指定webhook容器

    1）启动容器

    1. ~]# docker run  --name=webhook  --net=host -v /etc/localtime:/etc/localtime -v /data/prometheus/Dockerfile2/webkook/app.py:/src/app.py  -d  -e ROBOT_TOKEN=eb25f0c4-69ac-458c-af07-a6999beb05cd webhook:v1
    2.  
    3. # /etc/localtime   时间要一致
    4. # -e ROBOT_TOKEN= 指定机器人 key

    2）告警配置指定webhook

    1. ]# cat alertmanager.yaml-wx 
    2.  
    3. global:
    4.        resolve_timeout: 5m
    5.  
    6. route:
    7.       group_by: ['alertname']
    8.       group_wait: 10s
    9.       group_interval: 10s
    10.       repeat_interval: 1m
    11.       receiver: 'web.hook'
    12.  
    13. receivers:
    14.     - name: 'web.hook'
    15.       webhook_configs:
    16.       - url: 'http://起容器的ip:5000'
    17.         send_resolved: true
    18. inhibit_rules:
    19.       - source_match:
    20.            severity: 'critical'
    21.         target_match:
    22.            severity: 'warning'
    23.         equal: ['alertname', 'dev', 'instance']

    注：如果转发不了记得把端口映射出来

  • 相关阅读:
    身份证OCR识别：黑科技助力证件信息快速提取
    【无标题】
    Windows 根据github上的环境需求，安装一个虚拟环境，安装cuda和torch
    [大家的项目] cargo-offline 命令
    pytorch深度学习实战15
    浅谈JMM和并发三大特性(volatile、MESI、内存屏障)
    Springboot毕设项目儿童医院问诊导诊系统aqy75（java+VUE+Mybatis+Maven+Mysql）
    6-9 共享后缀的链表 分数 15
    Nuxt服务端请求及获取Cookie
    Java学习之继承一
  • 原文地址：https://blog.csdn.net/kali_yao/article/details/126412479