• K8s Pod 创建埋点处理(Mutating Admission Webhook)

    写在前面

    • 工作中涉及到相关的知识
    • 在实际的生产中，我们可以有需求对 创建的资源做一些类似 埋点 相关的操作，比如添加一些 Pod 创建时的自定义验证逻辑，类似表单提交验证那样，或者希望对创建的资源对象进行加工，在比如给资源对象添加对应的 zone 标签，涉及到 SC 相关的标签，或者根据命名空间动态织入亲和性和拓扑相关约束,添加一些 卷
    • 上面的这些需求我们可以通过 k8s Admission Webhook来完成，博文为查阅资料整理笔记，内容为涉及
    • 一个大佬写好的 自定义准入控制器 Demo 学习
    • 理解不足小伙伴帮忙指正

    对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》

    学习之前，建议看看这两篇官网的文章：

    准入控制器参考:

    https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

    Kubernetes 准入控制器指南:

    https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

    学习的项目文章：

    https://didil.medium.com/building-a-kubernetes-mutating-admission-webhook-7e48729523ed

    翻译版本：

    https://cloudnative.to/blog/mutating-admission-webhook/

    通过 Admission Webhook 为每个创建的 Pod(打了指定标签) 添加一个 CM ，以卷的方式挂载，类似为 每个命名空间 中的 Pod 自动挂载当前命名默认生成 SA 的 token 一样。

    克隆一下作者的项目：

    https://github.com/didil/k8s-hello-mutating-webhook

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook]
└─$git clone  https://github.com/didil/k8s-hello-mutating-webhook.git

    • 1
    • 2

    确认一下 对应的 Webhook 钩子对应的镜像能不能使用

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$docker pull quay.io/didil/hello-webhook:0.1.9
0.1.9: Pulling from didil/hello-webhook
df20fa9351a1: Pull complete
c5ef763ec908: Pull complete
242c22b4c8b8: Pull complete
f201599daacc: Pull complete
Digest: sha256:8c6ef414e4df46f15b9b63f83bd24f7c6a2cc5b5b937c53f0811fd509969ad05
Status: Downloaded newer image for quay.io/didil/hello-webhook:0.1.9
quay.io/didil/hello-webhook:0.1.9

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10

    学习的环境

    ┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get node
NAME                          STATUS   ROLES           AGE    VERSION
vms100.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms101.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms102.liruilongs.github.io   Ready    control-plane   292d   v1.25.1
vms103.liruilongs.github.io   Ready    <none>          292d   v1.25.1
vms105.liruilongs.github.io   Ready    <none>          292d   v1.25.1
vms106.liruilongs.github.io   Ready    <none>          292d   v1.25.1
┌──[root@vms100.liruilongs.github.io]-[~]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11

    看下作者的 yaml 文件，可以看到，当前作者使用 kustomization 来管理 k8s yaml 资源文件, kustomize 是一个 生成 k8s 资源 yaml 文件的插件，支持继承组合等一些面向对象思维方面的 yaml 文件生成

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$ls
csr  deployment  other
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$tree .
.
├── csr
│   ├── csr-job.yaml
│   ├── csr-rolebinding.yaml
│   ├── csr-role.yaml
│   ├── csr-sa.yaml
│   └── kustomization.yaml
├── deployment
│   ├── deployment.yaml
│   └── kustomization.yaml
└── other
    ├── configmap.yaml
    ├── kustomization.yaml
    ├── service.yaml
    └── webhookconf.yaml

3 directories, 11 files
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24

    需要安装 kustomize ,嫌麻烦当然也可以直接看

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s]
└─$kustomize version
{Version:kustomize/v4.5.7 GitCommit:56d82a8378dfc8dc3b3b1085e5a6e67b82966bd7 BuildDate:2022-08-02T16:35:54Z GoOs:linux GoArch:amd64}


    • 1
    • 2
    • 3
    • 4

    这里分别看下作者的三个目录里面放了什么

    deployment 目录主要为注入逻辑的工作负载hello-webhook，这里需要注意的是挂载了一个证书文件，由于 Webhook 必须通过 HTTPS 提供，因此我们需要为服务器提供适当的自签名 CA 签名。证书的公用名 （CN） 必须与 Kubernetes API 服务器使用的服务器名称匹配。

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$ls
deployment.yaml  kustomization.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$kubectl kustomize ./
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-webhook
  name: hello-webhook-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-webhook
  template:
    metadata:
      labels:
        app: hello-webhook
    spec:
      containers:
      - image: quay.io/didil/hello-webhook:0.1.9
        name: hello-webhook
        ports:
        - containerPort: 8000
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
        volumeMounts:
        - mountPath: /tls
          name: hello-tls-secret
          readOnly: true
      volumes:
      - name: hello-tls-secret
        secret:
          secretName: hello-tls-secret
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/deployment]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40

    下面为 webhook 的 核心逻辑,用于对 Pod 添加对应的 CM 卷 ,添加对应的操作记录，返回修改后的对象

    func (app *App) HandleMutate(w http.ResponseWriter, r *http.Request) {
 admissionReview := &admissionv1.AdmissionReview{}

 // read the AdmissionReview from the request json body
 err := readJSON(r, admissionReview)
 if err != nil {
  app.HandleError(w, r, err)
  return
 }

 // unmarshal the pod from the AdmissionRequest
 pod := &corev1.Pod{}
 if err := json.Unmarshal(admissionReview.Request.Object.Raw, pod); err != nil {
  app.HandleError(w, r, fmt.Errorf("unmarshal to pod: %v", err))
  return
 }

 // add the volume to the pod
 pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{
  Name: "hello-volume",
  VolumeSource: corev1.VolumeSource{
   ConfigMap: &corev1.ConfigMapVolumeSource{
    LocalObjectReference: corev1.LocalObjectReference{
     Name: "hello-configmap",
    },
   },
  },
 })

 // add volume mount to all containers in the pod
 for i := 0; i < len(pod.Spec.Containers); i++ {
  pod.Spec.Containers[i].VolumeMounts = append(pod.Spec.Containers[i].VolumeMounts, corev1.VolumeMount{
   Name:      "hello-volume",
   MountPath: "/etc/config",
  })
 }

 containersBytes, err := json.Marshal(&pod.Spec.Containers)
 if err != nil {
  app.HandleError(w, r, fmt.Errorf("marshall containers: %v", err))
  return
 }

 volumesBytes, err := json.Marshal(&pod.Spec.Volumes)
 if err != nil {
  app.HandleError(w, r, fmt.Errorf("marshall volumes: %v", err))
  return
 }

 // build json patch
 patch := []JSONPatchEntry{
  JSONPatchEntry{
   OP:    "add",
   Path:  "/metadata/labels/hello-added",
   Value: []byte(`"OK"`),
  },
  JSONPatchEntry{
   OP:    "replace",
   Path:  "/spec/containers",
   Value: containersBytes,
  },
  JSONPatchEntry{
   OP:    "replace",
   Path:  "/spec/volumes",
   Value: volumesBytes,
  },
 }

 patchBytes, err := json.Marshal(&patch)
 if err != nil {
  app.HandleError(w, r, fmt.Errorf("marshall jsonpatch: %v", err))
  return
 }

 patchType := admissionv1.PatchTypeJSONPatch

 // build admission response
 admissionResponse := &admissionv1.AdmissionResponse{
  UID:       admissionReview.Request.UID,
  Allowed:   true,
  Patch:     patchBytes,
  PatchType: &patchType,
 }

 respAdmissionReview := &admissionv1.AdmissionReview{
  TypeMeta: metav1.TypeMeta{
   Kind:       "AdmissionReview",
   APIVersion: "admission.k8s.io/v1",
  },
  Response: admissionResponse,
 }

 jsonOk(w, &respAdmissionReview)
}

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
    • 63
    • 64
    • 65
    • 66
    • 67
    • 68
    • 69
    • 70
    • 71
    • 72
    • 73
    • 74
    • 75
    • 76
    • 77
    • 78
    • 79
    • 80
    • 81
    • 82
    • 83
    • 84
    • 85
    • 86
    • 87
    • 88
    • 89
    • 90
    • 91
    • 92
    • 93
    • 94

    上述代码主要做了如下事情：

    • 将来自 Http 请求中的 AdmissionReview json 输入反序列化。
    • 读取 Pod 的 spec 信息。
    • 将 hello-configmap 作为数据源，添加 hello-volume 卷到 Pod。
    • 挂载卷至 Pod 容器中。
    • 以 JSON PATCH 的形式记录变更信息，包括卷的变更，卷挂载信息的变更。顺道为容器添加一个“hello-added=true”的标签。
    • 构建 json 格式的响应结果，结果中包含了这次请求中的被修改的部分。

    other 目录主要放了 需要动态织入的 CM，以卷的方式使用，以及 MutatingWebhookConfiguration 准入控制器的定义，以及 webhook 的 SVC(SVC 名字太长，没办法生成证书，需要修改一下)

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$kubectl kustomize .
apiVersion: v1
data:
  hello.txt: "\n /$$$$$$$$ /$$   /$$ /$$$$$$  /$$$$$$        /$$$$$$  /$$$$$$        /$$
    \  /$$  /$$$$$$   /$$$$$$ \n|__  $$__/| $$  | $$|_  $$_/ /$$__  $$      |_  $$_/
    /$$__  $$      | $$  /$$/ /$$__  $$ /$$__  $$\n   | $$   | $$  | $$  | $$  | $$
    \ \\__/        | $$  | $$  \\__/      | $$ /$$/ | $$  \\ $$| $$  \\__/\n   | $$
    \  | $$$$$$$$  | $$  |  $$$$$$         | $$  |  $$$$$$       | $$$$$/  |  $$$$$$/|
    \ $$$$$$ \n   | $$   | $$__  $$  | $$   \\____  $$        | $$   \\____  $$      |
    $$  $$   >$$__  $$ \\____  $$\n   | $$   | $$  | $$  | $$   /$$  \\ $$        |
    $$   /$$  \\ $$      | $$\\  $$ | $$  \\ $$ /$$  \\ $$\n   | $$   | $$  | $$ /$$$$$$|
    \ $$$$$$/       /$$$$$$|  $$$$$$/      | $$ \\  $$|  $$$$$$/|  $$$$$$/\n   |__/
    \  |__/  |__/|______/ \\______/       |______/ \\______/       |__/  \\__/ \\______/
    \ \\______/ \n                                                                                                  \n
    \                                                                                                 \n
    \                                                                                                 \n"
kind: ConfigMap
metadata:
  name: hello-configmap
---
apiVersion: v1
kind: Service
metadata:
  name: hello-webhook-service
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 8000
  selector:
    app: hello-webhook
  type: ClusterIP
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: hello-webhook.leclouddev.com
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    service:
      name: hello-webhook-service
      namespace: default
      path: /mutate
  name: hello-webhook.leclouddev.com
  objectSelector:
    matchLabels:
      hello: "true"
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 10
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
    • 63
    • 64
    • 65

    简单分析一下 MutatingWebhookConfiguration ,主要的配置 webhooks 配置

    apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: hello-webhook.leclouddev.com
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig: #指定了客户端配置，用于指定 Webhook 的服务信息。
    service:
      name: hello-webhook-service
      namespace: default
      path: /mutate
  name: hello-webhook.leclouddev.com #指定了 Webhook 的名称
  objectSelector: # 指定了对象选择器，用于选择要应用 Webhook 的对象。
    matchLabels:
      hello: "true"
  rules: #指定了 Webhook 的规则
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 10 #指定了 Webhook 的超时时间，此处为 10 秒。

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29

    这里可以看到当前准入控制器 webhook 只处理 打了标签 hello=true 的 pod

    csr 目录为权限，生成证书的 Job，SA 以及 通过 SA 添加对应的集群权限

    生成的相关证书,用于 K8s Webhook 通信，相关项目地址：

    https://github.com/didil/k8s-webhook-cert-manager

    当前作者的 csr-job.yaml 封装的 pod 比较旧，shell 脚本需要调整，所有我们不需要，直接本地生成 证书和对应的秘密

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$ls
csr-job.yaml  csr-rolebinding.yaml  csr-role.yaml  csr-sa.yaml  kustomization.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl kustomize  ./
apiVersion: v1
kind: ServiceAccount
metadata:
  name: webhook-cert-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: webhook-cert-cluster-role
rules:
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  verbs:
  - get
  - create
  - patch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - create
  - get
  - delete
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/approval
  verbs:
  - update
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - get
  - patch
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: webhook-cert-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: webhook-cert-cluster-role
subjects:
- kind: ServiceAccount
  name: webhook-cert-sa
  namespace: default
---
apiVersion: batch/v1
kind: Job
metadata:
  name: webhook-cert-setup
spec:
  backoffLimit: 3
  template:
    spec:
      containers:
      - args:
        - --service
        - hello-webhook-service
        - --webhook
        - hello-webhook.leclouddev.com
        - --secret
        - hello-tls-secret
        - --namespace
        - default
        command:
        - ./generate_certificate.sh
        image: quay.io/didil/k8s-webhook-cert-manager:0.13.19-1-a
        name: webhook-cert-setup
      restartPolicy: OnFailure
      serviceAccountName: webhook-cert-sa
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
    • 63
    • 64
    • 65
    • 66
    • 67
    • 68
    • 69
    • 70
    • 71
    • 72
    • 73
    • 74
    • 75
    • 76
    • 77
    • 78
    • 79
    • 80
    • 81
    • 82
    • 83
    • 84
    • 85
    • 86
    • 87
    • 88
    • 89
    • 90
    • 91

    部署项目

    在部署之前，我们需要改一下命名空间,创建一个新的命名空间，mutating-webhook

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl create  ns mutating-webhook
namespace/mutating-webhook created

    • 1
    • 2
    • 3

    然后修改一下每个目录的 kustomization.yaml 文件,类似下面这样，添加 namespace: mutating-webhook

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/other]
└─$cat kustomization.yaml
namespace: mutating-webhook
resources:
- configmap.yaml
- service.yaml
- webhookconf.yaml

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

    然后还需要修改一下证书生成的的命名空间

    ./generate_certificate.sh  --service hello-webhook-service --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace mutating-webhook

    • 1

    之后就可以部署了

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ./
serviceaccount/webhook-cert-sa created
clusterrole.rbac.authorization.k8s.io/webhook-cert-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/webhook-cert-cluster-role-binding created
job.batch/webhook-cert-setup created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ../other/
configmap/hello-configmap created
service/hello-webhook-service created
mutatingwebhookconfiguration.admissionregistration.k8s.io/hello-webhook.leclouddev.com created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl apply  -k ../deployment/
deployment.apps/hello-webhook-deployment created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl config  set-context --current --namespace mutating-webhook
Context "kubernetes-admin@kubernetes" modified.

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17

    webhook 一直没办法正常运行

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl get all
NAME                                            READY   STATUS              RESTARTS      AGE
pod/hello-webhook-deployment-7f599b95c4-pg9w5   0/1     ContainerCreating   0             55s
pod/webhook-cert-setup-v62rt                    0/1     CrashLoopBackOff    2 (15s ago)   79s

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/hello-webhook-service   ClusterIP   10.103.24.30   <none>        443/TCP   59s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-webhook-deployment   0/1     1            0           55s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-webhook-deployment-7f599b95c4   1         1         0       55s

NAME                           COMPLETIONS   DURATION   AGE
job.batch/webhook-cert-setup   0/1           79s        79s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19

    这里发现作者生成证书的 job 中脚本使用的 镜像中的 kubectl 版本太低了，没办法正常执行，所以我这么直接在的本地生成 证书，创建对应的 secret ,任然使用作者的 脚本generate_certificate.sh

    作者原来的证书签名请求对应的 API 资源对象使用的是bata 版本，现在已经更新为正式版本，需要替换一下,其他部分逻辑也需要调整，下面为调整后的

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$cat generate_certificate.sh
#!/usr/bin/env sh

set -e

usage() {
  cat <<EOF
Generate certificate suitable for use with any Kubernetes Mutating Webhook.
This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with any Kubernetes Mutating Webhook service pod.
This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.
The server key/cert k8s CA cert are stored in a k8s secret.
usage: ${0} [OPTIONS]
The following flags are required.
    --service          Service name of webhook.
    --webhook          Webhook config name.
    --namespace        Namespace where webhook service and secret reside.
    --secret           Secret name for CA certificate and server certificate/key pair.
The following flags are optional.
    --webhook-kind     Webhook kind, either MutatingWebhookConfiguration or
                       ValidatingWebhookConfiguration (defaults to MutatingWebhookConfiguration)
EOF
  exit 1
}

while [ $# -gt 0 ]; do
  case ${1} in
      --service)
          service="$2"
          shift
          ;;
      --webhook)
          webhook="$2"
          shift
          ;;
      --secret)
          secret="$2"
          shift
          ;;
      --namespace)
          namespace="$2"
          shift
          ;;
      --webhook-kind)
          kind="$2"
          shift
          ;;
      *)
          usage
          ;;
  esac
  shift
done

[ -z "${service}" ] && echo "ERROR: --service flag is required" && exit 1
[ -z "${webhook}" ] && echo "ERROR: --webhook flag is required" && exit 1
[ -z "${secret}" ] && echo "ERROR: --secret flag is required" && exit 1
[ -z "${namespace}" ] && echo "ERROR: --namespace flag is required" && exit 1

fullServiceDomain="${service}.${namespace}.svc"

# THE CN has a limit of 64 characters. We could remove the namespace and svc
# and rely on the Subject Alternative Name (SAN), but there is a bug in EKS
# that discards the SAN when signing the certificates.
#
# https://github.com/awslabs/amazon-eks-ami/issues/341
if [ ${#fullServiceDomain} -gt 64 ] ; then
  echo "ERROR: common name exceeds the 64 character limit: ${fullServiceDomain}"
  exit 1
fi

if [ ! -x "$(command -v openssl)" ]; then
  echo "ERROR: openssl not found"
  exit 1
fi

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> "${tmpdir}/csr.conf"
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${fullServiceDomain}
DNS.4 = ${fullServiceDomain}.cluster.local
EOF
echo "/CN=${fullServiceDomain}"
openssl genrsa -out "${tmpdir}/server-key.pem" 2048
#openssl req -new -key "${tmpdir}/server-key.pem" -subj "/CN=${fullServiceDomain}" -out "${tmpdir}/server.csr" -config "${tmpdir}/csr.conf"
openssl req -new -key "${tmpdir}/server-key.pem" -subj "/CN=system:node:${fullServiceDomain};/O=system:nodes" -out "${tmpdir}/server.csr" -config "${tmpdir}/csr.conf"
set +e
# clean-up any previously created CSR for our service. Ignore errors if not present.
if kubectl delete csr "${csrName}"; then
    echo "WARN: Previous CSR was found and removed."
fi
set -e

# create server cert/key CSR and send it to k8s api
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  #signerName: kubernetes.io/kube-apiserver-client
  signerName: kubernetes.io/kubelet-serving
  groups:
  - system:authenticated
  request: $(base64 < "${tmpdir}/server.csr" | tr -d '\n')
  usages:
  - server auth
  - digital signature
  - key encipherment
EOF

set +e
# verify CSR has been created
while true; do
  if kubectl get csr "${csrName}"; then
      echo "CertificateSigningRequest create succsee"
      break
  fi
done
set -e

# approve and fetch the signed certificate . !! not working with k8s 1.19.1, running the command separately outside of the container / node
set +e
while true; do
  if kubectl certificate approve "${csrName}"; then
     echo "${csrName} certificate approve"
     break
  fi
done

set -e

set +e
# verify certificate has been signed
i=1
while [ "$i" -ne 10 ]
do
  serverCert=$(kubectl get csr "${csrName}" -o jsonpath='{.status.certificate}')
  if [ "${serverCert}" != '' ]; then
      break
  fi
  sleep 5
  i=$((i + 1))
done

set -e
if [ "${serverCert}" = '' ]; then
  echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
  exit 1
fi

echo "${serverCert}" | openssl base64 -d -A -out "${tmpdir}/server-cert.pem"

# create the secret with CA cert and server cert/key
kubectl create secret tls "${secret}" \
      --key="${tmpdir}/server-key.pem" \
      --cert="${tmpdir}/server-cert.pem" \
      --dry-run -o yaml |
  kubectl -n "${namespace}" apply -f -

#caBundle=$(base64 < /run/secrets/kubernetes.io/serviceaccount/ca.crt  | tr -d '\n')
caBundle=$(cat ${tmpdir}/server-cert.pem)
set +e
# Patch the webhook adding the caBundle. It uses an `add` operation to avoid errors in OpenShift because it doesn't set
# a default value of empty string like Kubernetes. Instead, it doesn't create the caBundle key.
# As the webhook is not created yet (the process should be done manually right after this job is created),
# the job will not end until the webhook is patched.
while true; do
  echo "INFO: Trying to patch webhook adding the caBundle."
  if kubectl patch "${kind:-mutatingwebhookconfiguration}" "${webhook}" --type='json' -p "[{'op': 'add', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${serverCert}'}]"; then
      break
  fi
  echo "INFO: webhook not patched. Retrying in 5s..."
  sleep 5
done
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
    • 63
    • 64
    • 65
    • 66
    • 67
    • 68
    • 69
    • 70
    • 71
    • 72
    • 73
    • 74
    • 75
    • 76
    • 77
    • 78
    • 79
    • 80
    • 81
    • 82
    • 83
    • 84
    • 85
    • 86
    • 87
    • 88
    • 89
    • 90
    • 91
    • 92
    • 93
    • 94
    • 95
    • 96
    • 97
    • 98
    • 99
    • 100
    • 101
    • 102
    • 103
    • 104
    • 105
    • 106
    • 107
    • 108
    • 109
    • 110
    • 111
    • 112
    • 113
    • 114
    • 115
    • 116
    • 117
    • 118
    • 119
    • 120
    • 121
    • 122
    • 123
    • 124
    • 125
    • 126
    • 127
    • 128
    • 129
    • 130
    • 131
    • 132
    • 133
    • 134
    • 135
    • 136
    • 137
    • 138
    • 139
    • 140
    • 141
    • 142
    • 143
    • 144
    • 145
    • 146
    • 147
    • 148
    • 149
    • 150
    • 151
    • 152
    • 153
    • 154
    • 155
    • 156
    • 157
    • 158
    • 159
    • 160
    • 161
    • 162
    • 163
    • 164
    • 165
    • 166
    • 167
    • 168
    • 169
    • 170
    • 171
    • 172
    • 173
    • 174
    • 175
    • 176
    • 177
    • 178
    • 179
    • 180
    • 181
    • 182
    • 183
    • 184
    • 185
    • 186
    • 187
    • 188
    • 189
    • 190
    • 191
    • 192
    • 193
    • 194

    生成证书，并且 创建 csr secret ，同时更新 mutatingwebhookconfiguration 的 caBundle 字段

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$./generate_certificate.sh  --service webhook-svc --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace mutating-webhook
creating certs in tmpdir /tmp/tmp.Di367dgyMz
/CN=webhook-svc.mutating-webhook.svc
Generating RSA private key, 2048 bit long modulus
........................................+++
.........+++
e is 65537 (0x10001)
Error from server (NotFound): certificatesigningrequests.certificates.k8s.io "webhook-svc.mutating-webhook" not found
certificatesigningrequest.certificates.k8s.io/webhook-svc.mutating-webhook created
NAME                           AGE   SIGNERNAME                      REQUESTOR          REQUESTEDDURATION   CONDITION
webhook-svc.mutating-webhook   0s    kubernetes.io/kubelet-serving   kubernetes-admin   <none>              Pending
CertificateSigningRequest create succsee
certificatesigningrequest.certificates.k8s.io/webhook-svc.mutating-webhook approved
webhook-svc.mutating-webhook certificate approve
W1115 17:25:40.403890   66211 helpers.go:663] --dry-run is deprecated and can be replaced with --dry-run=client.
secret/hello-tls-secret created
INFO: Trying to patch webhook adding the caBundle.
mutatingwebhookconfiguration.admissionregistration.k8s.io/hello-webhook.leclouddev.com patched

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19

    如果 命名空间或者 svc 名字太长的话，会报下面的错，需要调整短一点

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$./generate_certificate.sh  --service hello-webhook-service --webhook hello-webhook.leclouddev.com --secret hello-tls-secret --namespace k8s-hello-mutating-webhook
creating certs in tmpdir /tmp/tmp.GcNh5TwKXP
/CN=hello-webhook-service.k8s-hello-mutating-webhook.svc
Generating RSA private key, 2048 bit long modulus
....+++
...................................................+++
e is 65537 (0x10001)
problems making Certificate Request
140682165290896:error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long:a_mbstr.c:158:maxsize=64

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10

    测试结果

    创建指定标签的 Pod ,自动挂载 CM

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl run busybox-1 --image=busybox  --restart=Never -l=app=busybox,hello=true -- sleep 3600
pod/busybox-1 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-1 -it -- sh -c "ls /etc/config/hello.txt"
/etc/config/hello.txt
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-1 -it -- sh -c "cat /etc/config/hello.txt"

 /$$$$$$$$ /$$   /$$ /$$$$$$  /$$$$$$        /$$$$$$  /$$$$$$        /$$   /$$  /$$$$$$   /$$$$$$
|__  $$__/| $$  | $$|_  $$_/ /$$__  $$      |_  $$_/ /$$__  $$      | $$  /$$/ /$$__  $$ /$$__  $$
   | $$   | $$  | $$  | $$  | $$  \__/        | $$  | $$  \__/      | $$ /$$/ | $$  \ $$| $$  \__/
   | $$   | $$$$$$$$  | $$  |  $$$$$$         | $$  |  $$$$$$       | $$$$$/  |  $$$$$$/|  $$$$$$
   | $$   | $$__  $$  | $$   \____  $$        | $$   \____  $$      | $$  $$   >$$__  $$ \____  $$
   | $$   | $$  | $$  | $$   /$$  \ $$        | $$   /$$  \ $$      | $$\  $$ | $$  \ $$ /$$  \ $$
   | $$   | $$  | $$ /$$$$$$|  $$$$$$/       /$$$$$$|  $$$$$$/      | $$ \  $$|  $$$$$$/|  $$$$$$/
   |__/   |__/  |__/|______/ \______/       |______/ \______/       |__/  \__/ \______/  \______/



    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19

    没有标签的没有自动挂载

    ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl run busybox-2 --image=busybox --restart=Never -l=app=busybox -- sleep 3600
pod/busybox-2 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl get pod -l=app=busybox -L=hello-added
NAME        READY   STATUS    RESTARTS   AGE   HELLO-ADDED
busybox-1   1/1     Running   0          30m   OK
busybox-2   1/1     Running   0          28s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$kubectl exec busybox-2 -it -- sh -c "ls /etc/config/hello.txt"
ls: /etc/config/hello.txt: No such file or directory
command terminated with exit code 1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s-hello-mutating-webhook/k8s-hello-mutating-webhook/k8s/csr]
└─$

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14

    博文部分内容参考

    © 文中涉及参考链接内容版权归原作者所有，如有侵权请告知，这是一个开源项目，如果你认可它，不要吝啬星星哦 😃

    https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#service-reference

    https://kubernetes.io/zh-cn/docs/reference/kubernetes-api/extend-resources/mutating-webhook-configuration-v1/

    https://cloudnative.to/blog/mutating-admission-webhook/

    https://github.com/didil/k8s-hello-mutating-webhook

    https://didil.medium.com/building-a-kubernetes-mutating-admission-webhook-7e48729523ed

    © 2018-2023 liruilonger@gmail.com, All rights reserved. 保持署名-非商用-相同方式共享(CC BY-NC-SA 4.0)

  • 相关阅读:
    Netty学习(二)概述+EventLoop+Channel+ByteBuf
    Python:实现linear regression线性回归算法(附完整源码)
    Docker安装Seata
    web自动化测试工具之Selenium的使用
    openCV第三篇
    Python3 下载、安装教程，附详细图解
    RISC-V 编译环境搭建：riscv-gnu-toolchain 和 riscv-tools
    SpringCloud Alibaba核心组件Nacos【服务多级存储模型&配置集群】第2章
    github 终端克隆操作，以及对 https/ssh 的理解
    WebSocket理解和使用
  • 原文地址：https://blog.csdn.net/sanhewuyang/article/details/134426489