Kibana使用Watcher监控服务日志并发送飞书报警(Markdown)

Watcher是什么

Kibana Watcher 是 Elasticsearch 的监控和告警工具，它允许你设置和管理告警规则以监控 Elasticsearch 数据和集群的状态。Kibana Watcher 可以监测各种指标和数据，然后在满足特定条件时触发警报。它提供了一种强大的方式来实时监控 Elasticsearch 数据、集群性能、日志和事件，以及其他关键指标。

构造飞书报警内容

创建飞书群聊并添加飞书报警机器人, 获取Webhook地址

构造请求体

{
    "msg_type": "interactive",
    "card": {
        "header": {
            "template": "red",
            "title": {
                "tag": "plain_text",
                "content": "接口请求预警(20m)"
            }
        },
        "config": {
            "wide_screen_mode": true,
            "enable_forward": true
        },
        "elements": [
            {
                "tag": "markdown",
                "content": "**接口:** /lastVersion\n**预警值:** 1000\n**请求次数:** 10000"
            }
        ]
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

添加Watcher

入口: Stack Management -> Watcher -> Create
2种方式:
Create threshold alert: 一般用于简单的指标监控, 及Index数量统计
Create advanced watch: 高级预警, 用于创建更复杂和灵活的监控规则。

此时我们选Create advanced watch

Name为Watcher名称, 用于标识
ID为自动生成, 也可编辑
Watch JSON为具体配置, 示例如下:

{
  "trigger": { // 触发时间
    "schedule": {
      "interval": "10m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [ // 索引
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "track_total_hits": true, // 精确查找个数
          "query": { // 查询近30min的/lastVersion请求次数
            "bool": {
              "must": {
                "match": {
                  "message": "/lastVersion"
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "from": "{{ctx.trigger.scheduled_time}}||-30m",
                    "to": "{{ctx.trigger.triggered_time}}"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": { // 触发条件
    "compare": {
      "ctx.payload.hits.total": { // 总数大于25w触发action
        "gte": 250000
      }
    }
  },
  "actions": {
    "feishu_webhook": { // 飞书报警
      "webhook": {
        "scheme": "https",
        "host": "open.feishu.cn",
        "port": 443,
        "method": "post",
        "path": "/open-apis/bot/v2/hook/**********",
        "params": {},
        "headers": {},
        "body": """
            {
                "msg_type": "interactive",
                "card": {
                    "header": {
                        "template": "red",
                        "title": {
                            "tag": "plain_text",
                            "content": "接口请求预警(30m)"
                        }
                    },
                    "config": {
                        "wide_screen_mode": true,
                        "enable_forward": true
                    },
                    "elements": [
                        {
                            "tag": "markdown",
                            "content": "**接口:** /lastVersion\n**预警值:** 200000\n**请求次数:** {{ctx.payload.hits.total}}"
                        }
                    ]
                }
            }
        """
      }
    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

发送结果

如果报警的内容需要循环打印, 如TOP5, 则使用以下表达式:

"body": """
   {
        "msg_type": "interactive",
        "card": {
            "header": {
                "template": "red",
                "title": {
                    "tag": "plain_text",
                    "content": "接口请求统计"
                }
            },
            "config": {
                "wide_screen_mode": true,
                "enable_forward": true
            },
            "elements": [
                {
                    "tag": "markdown",
                    "content": "**调用结果:**{{#ctx.payload.aggregations.top_5.buckets}}\n{{key}} ({{doc_count}}){{/ctx.payload.aggregations.top_5.buckets}}"
                }
            ]
        }
    }
"""
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24