【云原生 | Kubernetes 系列】---Prometheus的服务发现机制

1. Prometheus的服务发现机制

prometheus默认是采用pull方式定时去目标拉取数据,每个被抓取目标向prometheus提供一个http接口,prometheus需要知道向哪个地址拉取数据,通过prometheus.yml中scrape_configs定义的job来实现目标的定义.这样就无法感知到新加入的变更信息.

通过daemonset实现在k8s环境中,当有新节点加入node-exporter和cadvisor的自动安装.
如果是传统方式,需要手动将新地址加入,并重启prometheus.
动态服务发现能够自动发现集群中的新端点,并加入配置中,通过服务发现,prometheus能够查询到需要监控的Target列表,然后轮询这个Target获取监控数据.

添加数据源的方法有很多种,常见的有静态配置和服务发现配置.
prometheus支持多种服务发现,常见的有:

kubernetes_sd_configs: kubernetes服务发现,让prometheus动态发现kubernetes中被监控的目标
static_configs: 静态服务发现,基于prometheus配置文件指定监控目标
dns_sd_configs: DNS服务发现监控目标
consul_sd_configs: Consul服务发现,基于consul服务动态发现监控目标
file_sd_configs: 基于指定的文件事先服务发现,prometheus定时读取文件如果发生变化则加载新目标

2. Prometheus 标签重写

prometheus的relabeling(标签重写)可以在抓取到目标实例之前把目标实例的元数据标签动态修改,动态添加或覆盖.

Prometheus在加载Target成功之后,在Taret实例中,都包含一些Metadata标签信息,默认标签有:

标签	含义
__address__	以:格式显示targets的地址
__scheme__	采用的目标服务地址的Scheme形式,HTTP或者HTTPS
__metrics_path__	采用的目标服务的访问路径

修改标签的时机:

relabel_configs: 在抓取数据之前,可以使用relabel_configs添加一些标签,也可以值采集特定目标或过滤目标
metric_relabel_configs: 在抓取数据之后,可以使用metric_relabel_configs做最后的重新标记和过滤

请添加图片描述

    - job_name: 'kubernetes-node-cadvisor'  #job名称
      kubernetes_sd_configs:	#基于kubernetes_sd_configs实现服务发现
      - role:  endpoint			# 发现endpoint
      scheme: https			# 当前job使用的发现协议
      tls_config:			# 证书配置
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt		# 容器里的证书路径
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token	# 容器里的token路径
      relabel_configs:		# 抓取前重写修改标签配置
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] # 源标签
      - action: keep		# action定义了relabel的具体动作.action支持多种
        regex: default;kubernetes;https   # 通过正则匹配条件
        #在default命名空间service名是kubernetes,并且使用https协议访问一旦条件成立,只保留匹配成功的数据(其实就是api-server)
1
2
3
4
5
6
7
8
9
10
11
12

2.1 label

source_labels: 源标签,没有经过relabel处理之前的名字
target_labels: 通过action处理之后的新标签名字
regex: 正则表达式,匹配源标签
replacement: 通过分组替换后标签(target_label)对应的值

2.2 action 详解

标签	含义
replace	替换标签值,根据regex正则匹配到源标签的值,使用replacement来引用表达式匹配的分组
keep	满足regex正则条件的实例进行采集,把source_labels中没有匹配到regex正则内容的target实例丢弃,只采集匹配成功的实例
drop	满足regex正则匹配条件的不采集,把source_labels中匹配到regex正则内容的Target丢弃,只采集没有匹配到的实例
Hashmod	使用Hashmod计算source_labels的Hash值并进行对比,基于自定义的模数取模,以实现对目标进行分类,重新赋值等功能
labelmap	匹配regex所有标签名称,然后赋值匹配标签的值进行分组,通过replacement分组引用替代
labelkeep	匹配regex所有标签名称,其他不匹配的标签都将从标签集中删除
labeldrop	匹配regex所有标签名称,其他匹配的标签都将从标签集中删除

2.3 发现类型

node
service
pod
endpoint
Endpointslice # 对endpoint进行切片
ingress

2.4 实现api-server监控

root@k8s-master-01:~# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.200.0.1   <none>        443/TCP   126d
root@k8s-master-01:~# kubectl get ep
NAME         ENDPOINTS                                                     AGE
kubernetes   192.168.31.101:6443,192.168.31.102:6443,192.168.31.103:6443   126d
1
2
3
4
5
6

通过监控api-server,判断集群的健康状况.

    - job_name: 'kubernetes-apiserver'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
1
2
3
4
5
6
7
8
9
10
11

2.5 监控kube-dns

创建kube-dns时需要再metadata中加入

      annotations:
        prometheus.io/port: "9153"
        prometheus.io/scrape: "true"
1
2
3

这样就能在kube-dns的注解中看到prometheus.io/port: 9153,prometheus.io/scrape: true
才能使得prometheus发现kube-dns

root@k8s-master-01:/opt/k8s-data/yaml/prometheus-files/case1/prometheus-files/case# kubectl describe svc kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
Labels:            addonmanager.kubernetes.io/mode=Reconcile
                   k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=CoreDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
1
2
3
4
5
6
7
8
9

通过configmap进行匹配
匹配任意namespace的endpoint

    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
      - role: endpoints			# 类型是endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]  # 匹配prometheus.io/scrape值
        action: keep
        regex: true		#如果是true则保留,继续往下匹配,否则就不进行保存
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)		# 匹配http和https
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)		# 至少1个字符任意长度
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2		# 将地址和端口两个值合并成 ip:port格式的__address__
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

2.5 cadvisor发现

确保所有节点都有cadvisor(daemonset控制)
服务端口8080

    - job_name: 'kubernetes-node-cadvisor'	# job名称
      kubernetes_sd_configs:		# 基于k8s发现
      - role:  node		#角色
      scheme: https	#协议
      tls_config:	# 证书配置
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:		#标签重写配置
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)	#匹配node_label
      - target_label: __address__					# 替换成
        replacement: kubernetes.default.svc:443		# 分组
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor  # $1是node_name
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

3. k8s外面的prometheus发现k8s中的服务

3.1 创建外部访问授权

授权外部访问k8s的授权

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
#apiVersion: rbac.authorization.k8s.io/v1beta1
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

创建授权

kubectl apply -f case4-prom-rbac.yaml
serviceaccount/prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
1
2
3
4

3.2 获取token

获取secrets

# kubectl get secrets -n monitoring 
NAME                     TYPE                                  DATA   AGE
default-token-cpcf8      kubernetes.io/service-account-token   3      25h
monitor-token-q8rcq      kubernetes.io/service-account-token   3      24h
prometheus-token-5stl8   kubernetes.io/service-account-token   3      114s
1
2
3
4
5

获取secrets的详细信息并保存

# kubectl describe secrets prometheus-token-5stl8 -n monitoring 
Name:         prometheus-token-5stl8
Namespace:    monitoring
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: prometheus
              kubernetes.io/service-account.uid: c4bd516b-bf0a-4427-8f40-447b5e4c2f6c

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1350 bytes
namespace:  10 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IjUxUzlHTUdYcUN2YnA2UU9ndEFSbEZJaUtySVFxdWJvVGc0TEFIR2NhbGcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb21ldGhldXMtdG9rZW4tNXN0bDgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImM0YmQ1MTZiLWJmMGEtNDQyNy04ZjQwLTQ0N2I1ZTRjMmY2YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOnByb21ldGhldXMifQ.D19ZpwmkdnEu5_jzN1oGqBRC2uu4euEoJT3U4WYRIjKIOGDqnB_cZrUUXkLBRPGDkR7VrYV8QZFyeeqswFZFa9nm-TEmJKAPGozL75GSZ5RSe8aMBm44P3mci5fx06jcio_qFUUkhgkWghkI84PYzjIrVhnN-LREVtn50lfpmzJLM26rCuQ2jaz7m3M-G8yrc_BI_nQ3N1MrgNAzupWOhh3n_bACXrY1SCcT2tVMLL8YLTtZVlCpaAUxTpaNMbCQVrXZQNyo0ZCLaqTAPbGe5fnDvfcago9U5Czfrihs3h5jJTjRF81yDE9-hVi8qtZPeIsGfzxIh75H5xH2WM924g
1
2
3
4
5
6
7
8
9
10
11
12
13
14

将token保存到二进制安装的prometheus服务器目录下

root@prometheus-2:/apps/prometheus# cat /apps/prometheus/k8s.token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjUxUzlHTUdYcUN2YnA2UU9ndEFSbEZJaUtySVFxdWJvVGc0TEFIR2NhbGcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InByb21ldGhldXMtdG9rZW4tNXN0bDgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoicHJvbWV0aGV1cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImM0YmQ1MTZiLWJmMGEtNDQyNy04ZjQwLTQ0N2I1ZTRjMmY2YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOnByb21ldGhldXMifQ.D19ZpwmkdnEu5_jzN1oGqBRC2uu4euEoJT3U4WYRIjKIOGDqnB_cZrUUXkLBRPGDkR7VrYV8QZFyeeqswFZFa9nm-TEmJKAPGozL75GSZ5RSe8aMBm44P3mci5fx06jcio_qFUUkhgkWghkI84PYzjIrVhnN-LREVtn50lfpmzJLM26rCuQ2jaz7m3M-G8yrc_BI_nQ3N1MrgNAzupWOhh3n_bACXrY1SCcT2tVMLL8YLTtZVlCpaAUxTpaNMbCQVrXZQNyo0ZCLaqTAPbGe5fnDvfcago9U5Czfrihs3h5jJTjRF81yDE9-hVi8qtZPeIsGfzxIh75H5xH2WM924g
1
2

3.3 配置监控api-server

在prometheus.yml追加以下内容

  - job_name: 'kubernetes-apiservers-monitor' 
    kubernetes_sd_configs: 
    - role: endpoints
      api_server: https://192.168.31.188:6443
      tls_config: 
        insecure_skip_verify: true  #跳过证书验证
      bearer_token_file: /apps/prometheus/k8s.token  # token地址 
    scheme: https 
    tls_config: 
      insecure_skip_verify: true 
    bearer_token_file: /apps/prometheus/k8s.token 
    relabel_configs: 
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] 
      action: keep 
      regex: default;kubernetes;https 
    - target_label: __address__ 
      replacement: 192.168.31.188:6443   # api集群地址
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

重启prometheus服务

root@prometheus-2:/apps/prometheus# systemctl restart prometheus.service
1

此时就可以在prometheus上看到apiservers的信息被采集到

请添加图片描述

3.4 配置监控node

  - job_name: 'kubernetes-nodes-monitor' 
    scheme: http 
    tls_config: 
      insecure_skip_verify: true 
    bearer_token_file: /apps/prometheus/k8s.token 
    kubernetes_sd_configs: 
    - role: node 
      api_server: https://192.168.31.188:6443 
      tls_config: 
        insecure_skip_verify: true 
      bearer_token_file: /apps/prometheus/k8s.token 
    relabel_configs: 
      - source_labels: [__address__] 
        regex: '(.*):10250' 
        replacement: '${1}:9100' 
        target_label: __address__ 
        action: replace 
      - action: labelmap 
        regex: __meta_kubernetes_node_label_(.+)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

重启prometheus服务后

请添加图片描述

3.5 配置监控Pod

  - job_name: 'kubernetes-pods-monitor' 
    kubernetes_sd_configs: 
    - role: pod 
      api_server: https://192.168.31.101:6443
      tls_config: 
        insecure_skip_verify: true 
      bearer_token_file: /apps/prometheus/k8s.token
    relabel_configs: 
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

在创建pod配置里需要加

  template:
    metadata:
      annotations:
        prometheus.io/port: "9180"
        prometheus.io/scrape: "true"
1
2
3
4
5

这样就能在prometheus中看到pod了

请添加图片描述

4. 基于Consul自动发现

Consul是分布式k/v数据存储集群,目前常用于服务注册和服务发现

4.1 consul的注册

向consul注册zookeep-1和zookeep-2数据

curl -X PUT -d '{"id": "zookeeper-1","name": "zookeeper-1","address": "192.168.31.121","port": 9100,"tags": ["zookeeper","node-exporter"],"checks": [{"http": "http://192.168.31.121:9100/","interval": "5s"}]}' http://192.168.31.121:8500/v1/agent/service/register
curl -X PUT -d '{"id": "zookeeper-2","name": "zookeeper-2","address": "192.168.31.122","port": 9100,"tags": ["zookeeper","node-exporter"],"checks": [{"http": "http://192.168.31.122:9100/","interval": "5s"}]}' http://192.168.31.121:8500/v1/agent/service/register
1
2

请添加图片描述

此时添加

curl -X PUT -d '{"id": "zookeeper-3","name": "zookeeper-3","address": "192.168.31.123","port": 9100,"tags": ["zookeeper","node-exporter"],"checks": [{"http": "http://192.168.31.123:9100/","interval": "5s"}]}' http://192.168.31.121:8500/v1/agent/service/register
1

请添加图片描述

这样就不需要每次都去修改配置文件再重启prometheus了.

4.2 consul的删除

curl --request PUT http://192.168.31.121:8500/v1/agent/service/deregister/cadvisor-1
curl --request PUT http://192.168.31.121:8500/v1/agent/service/deregister/zookeeper
1
2

5. 基于file的服务发现

5.1 编写sd_configs文件

通过prometheus加载本地文件,当本地文件发生变化,prometheus随之变化

mkdir -p /apps/prometheus/file_sd
cd /apps/prometheus/file_sd
vim sd_my_server.json
[
	{
		"targets": ["192.168.31.121:9100","192.168.31.122:9100","192.168.31.123:9100"]
	}
]
1
2
3
4
5
6
7
8

5.2 prometheus调用sd_configs

  - job_name: "file_sd_my_server"
    file_sd_configs:
      - files:
        - /apps/prometheus/file_sd/sd_my_server.json
        refresh_interval: 10s
1
2
3
4
5

重启prometheus

systemctl restart prometheus
1

请添加图片描述

修改配置,追加2台服务器

[
	{
		"targets": ["192.168.31.121:9100","192.168.31.122:9100","192.168.31.123:9100","192.168.31.111:9100","192.168.31.112:9100"]
	}
]
1
2
3
4
5

此时不再需要重启服务,等待刷新就可以看到新追加的服务

请添加图片描述

6. Dns服务发现

6.1 配置dns

这里用hosts代替

192.168.31.111 node-1 k8s-node-1
192.168.31.112 node-2 k8s-node-2
192.168.31.113 node-3 k8s-node-3
192.168.31.114 node-4 k8s-node-4
1
2
3
4

6.2 修改prometheus.yml

  - job_name: "file_sd_my_server"
    file_sd_configs:
      - files:
        - /apps/prometheus/file_sd/sd_my_server.json
        refresh_interval: 10s

  - job_name: "dns-server-monitor"
    metrics_path: /metrics
    dns_sd_configs:
    - names: ["k8s-node-1","k8s-node-2","k8s-node-3"]
      type: A
      port: 9100
1
2
3
4
5
6
7
8
9
10
11
12

6.3 重启prometheus服务

curl -X POST http://192.168.31.201:9090/-/reload
1

重启后dns节点发现
请添加图片描述

当变更hosts中的解析记录时

192.168.31.114 node-3 k8s-node-3
1

不用重启,节点信息自动变更.

请添加图片描述
但加减主机时是需要重启的

相关阅读:
【面试算法——动态规划 20】最长公共子序列&& 不相交的线
 优雅处理Golang中的异常
 Hive之函数
 WebGL笔记：使用鼠标绘制多个线条应用及绘制动感线性星座
 含文档+PPT+源码等]精品spring boot+MySQL客户关系管理系统vue[包运行成功]SpringBoot毕业设计程序设计项目源码
 Android开发基础——Fragment实践
 【自学记录】【Pytorch2.0深度学习从零开始学王晓华】第二章深度学习环境搭建
 前端工程化精讲第二十一课容器方案：从构建到部署，容器化方案的优势有哪些？
web前端期末大作业：基于HTML+CSS+JavaScript制作我的音乐网站（带设计报告）
《系列二》-- 1、BeanFactory.getBean 总览
原文地址：https://blog.csdn.net/qq_29974229/article/details/126645744