grafana+promethus+alertmanager+钉钉

1、在prometheus中配置预警规则

prometheus安装详见https://blog.csdn.net/u010924720/article/details/125915287

vim promethus.yml

rule_files:
  - "rules.yml"
1
2

注意：rule_files下面写绝对路径会有问题
1

promethus.yml同级目录下新建rules.yml

groups:
- name: node_alert
  rules:
  - alert: cpu_alert
    expr: 100 -avg(irate(node_cpu_seconds_total{mode="idle"}[1m])) by (instance)* 100 > 1
    for: 5m
    labels:
      level: warning
    annotations:
      description: "instance: {{ $labels.instance }} ,cpu usage is too high ! value: {{$value}}"
      summary:  "cpu usage is too high"
1
2
3
4
5
6
7
8
9
10
11

启动promethus时，加上rules.yml

docker run  -d \
  -p 9090:9090 \
  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml  \
  -v /opt/prometheus/rules.yml:/etc/prometheus/rules.yml \
  --name prometheus\
  prom/prometheus:v2.34.0
1
2
3
4
5
6

docker restart promethus后访问地址http://192.168.11.128:9090/rules

2、alertmanager和钉钉配置

1）创建钉钉机器人

群设置-智能群助手-添加机器人

2）安装钉钉和alertmanager

docker pull prom/alertmanager:v0.24.0
docker pull timonwong/prometheus-webhook-dingtalk
1
2

编写钉钉配置文件

vim /opt/webhook/config.yml

targets:
  webhook1:
    url: https://oapi.dingtalk.com/robot/send?access_token=518a7002228ebd28f205ac7ea2ax124064c25cd74e096f81f60bb7f9e9d3cde
    secret: SECa59327231011eb67fe6ec7e715f066b5d55c43bb2530fb699d75f8c55bdcd5bf
1
2
3
4

启动webhook

docker run  -d \
  -p 8060:8060 \
  -v /opt/webhook/config.yml:/etc/prometheus-webhook-dingtalk/config.yml  \
  --name webhook \
 timonwong/prometheus-webhook-dingtalk
1
2
3
4
5

编写告警配置文件

vim /opt/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
route:
  receiver: webhook
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 5m
  group_by: [alertname]  
  routes:  
  - receiver: webhook  
    group_wait: 10s  
receivers:  
- name: webhook  
  webhook_configs:  
  - url: http://192.168.11.128:8060/dingtalk/webhook1/send
    send_resolved: true
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

启动alertmanager

docker run -d --name alertmanager -p 9093:9093 -v /opt/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager:v0.24.0
1

访问页面http://192.168.11.128:9093/#/status

3、在prometheus中配置alertmanager

vim /opt/prometheus/prometheus.yml

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 192.168.11.128:9093
1
2
3
4
5

重启Prometheus，点击alerts结果如下

当预警触发后