k8s的部署管理以及prometheus相关监控

1. 安装运行prometheus和grafana
2. 编辑prometheus配置文件，增加采集prometheus自身数据的采集规则
3. grafana上导入prometheus相关图表的dashboard

前面三步主要根据https://blog.csdn.net/shnu_cdk/article/details/132182858?spm=1001.2014.3001.5506

4. 用Go编写一个prometheus exporter，包含prometheus四种指标类型

这一部分直接查看官方文档使用说明。

package main

import (
	"fmt"
	"math/rand"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	// 创建并注册指标
	counter := prometheus.NewCounter(prometheus.CounterOpts{
		Name: "my_counter",
		Help: "A counter metric",
	})
	prometheus.MustRegister(counter)

	gauge := prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "my_gauge",
		Help: "A gauge metric",
	})
	prometheus.MustRegister(gauge)

	histogram := prometheus.NewHistogram(prometheus.HistogramOpts{
		Name: "my_histogram",
		Help: "A histogram metric",
	})
	prometheus.MustRegister(histogram)

	summary := prometheus.NewSummary(prometheus.SummaryOpts{
		Name: "my_summary",
		Help: "A summary metric",
	})
	prometheus.MustRegister(summary)

	// 定期更新指标值
	go func() {
		for {
			counter.Inc()
			gauge.Set(rand.Float64() * 100)
			histogram.Observe(rand.Float64() * 100)
			summary.Observe(rand.Float64() * 100)
			time.Sleep(time.Second)
		}
	}()

	// 启动HTTP服务，暴露指标
	http.Handle("/metrics", promhttp.Handler())
	fmt.Println("Exporter is running on http://localhost:8080/metrics")
	http.ListenAndServe(":8080", nil)
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

5. prometheus能够正常采集第四步中的exporter，并在grafana上展示
   在yml配置文件上配置相对应的端口，这一部分和配置一个node差不多：

scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  
  - job_name: 'export'
    
    static_configs:
      - targets: ['192.168.60.177:8080']
        labels:
          instance: export
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

6. 了解alertmanager，知道alertmanager的webhook推送告警方式
   主要是要在prometheus上配置相应的端口

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['192.168.60.169:9093']
1
2
3
4
5

然后在altermanager里面配置相应的路由树

route:
  group_by: ['alertname']
  group_wait: 1s
  group_interval: 1s
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://192.168.60.188:8093/demo'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

接下来熟练操作k8s，我们采用搭建minikube来学习k8s
主要熟悉以下两个方面：

1. 通过kubectl创建资源

2. 使用client-go通过代码创建资源
   https://blog.51cto.com/daixuan/5184509