准入控制器(Admission Controller):ResourceQuota，ImagePolicyWebhook

目录

• 一.系统环境
• 二.前言
• 三.准入控制器简介
• 四.为什么需要准入控制器
• 五.启用/禁用ResourceQuota资源配额
  • 5.1 查看默认启用/禁用的准入控制器插件
  • 5.2 ResourceQuota资源配额示例
  • 5.3 禁用ResourceQuota
• 六.配置ImagePolicyWebhook准入控制器禁止使用后缀为latest的镜像
  • 6.1 搭建Webhook服务器
  • 6.2 配置kubernetes连接后端webhook服务器
  • 6.3 验证
• 七.总结

一.系统环境

本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。

服务器版本	docker软件版本	Kubernetes(k8s)集群版本	CPU架构
Ubuntu 18.04.5 LTS	Docker version 20.10.14	v1.22.2	x86_64

Kubernetes集群架构：k8scludes1作为master节点，k8scludes2，k8scludes3作为worker节点。

服务器	操作系统版本	CPU架构	进程	功能描述
k8scludes1/192.168.110.128	Ubuntu 18.04.5 LTS	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master节点
k8scludes2/192.168.110.129	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker节点
k8scludes3/192.168.110.130	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker节点

二.前言

在本文中，我们将探讨Kubernetes中的准入控制器（Admission Controller）。准入控制器是Kubernetes API server的一部分，负责处理来自外部的API请求。它们确保只有满足特定条件的请求才能访问集群资源。通过使用准入控制器，我们可以实现对集群资源的细粒度控制，从而提高集群的安全性和可靠性。

使用准入控制器（Admission Controller）的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。

三.准入控制器简介

当用户账户或者service account要执行一些操作的时候，首先要进行账号认证，认证通过之后，如果账号已经被授权相关资源，就可以进行相关操作了。有时授权之后也会有Admission control准入控制器(可以理解为一系列规则，可以启用或者关闭某个功能，准入控制器也支持一系列的webhook，在webhook里可以定义一系列规则)，满足Admission control准入控制器的规则就可以创建相关资源了。OPA Gatekeeper也是利用的准入控制器功能，详情请查看博客《OPA Gatekeeper：Kubernetes的策略和管理》。

准入控制器 是一段代码，它会在请求通过认证和鉴权之后、对象被持久化之前拦截到达 API 服务器的请求。

准入控制器可以执行验证（Validating） 和/或变更（Mutating） 操作。 变更（mutating）控制器可以根据被其接受的请求更改相关对象；验证（validating）控制器则不行。

准入控制器限制创建、删除、修改对象的请求。 准入控制器也可以阻止自定义动作，例如通过 API 服务器代理连接到 Pod 的请求。 准入控制器不会 （也不能）阻止读取（get、watch 或 list）对象的请求。

Kubernetes 1.30 中的准入控制器编译进 kube-apiserver 可执行文件，并且只能由集群管理员配置。 在该列表中，有两个特殊的控制器：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。 它们根据 API 中的配置， 分别执行变更和验证准入控制 Webhook。

准入控制过程分为两个阶段。第一阶段，运行变更准入控制器。第二阶段，运行验证准入控制器。 再次提醒，某些控制器既是变更准入控制器又是验证准入控制器。

如果两个阶段之一的任何一个控制器拒绝了某请求，则整个请求将立即被拒绝，并向最终用户返回错误。

最后，除了对对象进行变更外，准入控制器还可能有其它副作用：将相关资源作为请求处理的一部分进行变更。 增加配额用量就是一个典型的示例，说明了这样做的必要性。 此类用法都需要相应的回收或回调过程，因为任一准入控制器都无法确定某个请求能否通过所有其它准入控制器。

四.为什么需要准入控制器

Kubernetes 的若干重要功能都要求启用一个准入控制器，以便正确地支持该特性。 因此，没有正确配置准入控制器的 Kubernetes API 服务器是不完整的，它无法支持你所期望的所有特性。

五.启用/禁用ResourceQuota资源配额

5.1 查看默认启用/禁用的准入控制器插件

本文中的kube-apiserver是以pod的方式运行的，名字为：kube-apiserver-k8scludes1。

root@k8scludes1:~# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS        AGE
calico-kube-controllers-65898446b5-qd9q6   1/1     Running   21 (2d8h ago)   28d
calico-node-d6564                          1/1     Running   58 (2d8h ago)   59d
calico-node-jgvjb                          1/1     Running   67 (10h ago)    59d
calico-node-snkxp                          1/1     Running   58 (2d8h ago)   59d
coredns-7f6cbbb7b8-5gpjt                   1/1     Running   20 (2d8h ago)   28d
coredns-7f6cbbb7b8-xm6pl                   1/1     Running   20 (2d8h ago)   28d
etcd-k8scludes1                            1/1     Running   54 (2d8h ago)   58d
kube-apiserver-k8scludes1                  1/1     Running   15 (10h ago)    19d
kube-controller-manager-k8scludes1         1/1     Running   249 (10h ago)   59d
kube-proxy-464tx                           1/1     Running   52 (2d8h ago)   59d
kube-proxy-mkwpx                           1/1     Running   52 (2d8h ago)   59d
kube-proxy-vsc9q                           1/1     Running   53 (2d8h ago)   59d
kube-scheduler-k8scludes1                  1/1     Running   247 (10h ago)   59d

查看kube-apiserver的帮助信息。

root@k8scludes1:~# kubectl exec -it kube-apiserver-k8scludes1 -n kube-system -- kube-apiserver -h
The Kubernetes API server validates and configures data
for the api objects which include pods, services, replicationcontrollers, and
others. The API Server services REST operations and provides the frontend to the
     ......
      --goaway-chance float                                                                                                                                                                              
                To prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affected, and the client will
                reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single
                apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.
      --livez-grace-period duration                                                                                                                                                                      
                This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has
                elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.
     
      --vmodule moduleSpec                                                                                                                                                                               
                comma-separated list of pattern=N settings for file-filtered logging

查看默认启用/禁止的准入控制器插件有哪些？可以看到默认启用的准入控制器插件有：CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook。

root@k8scludes1:~# kubectl exec -it kube-apiserver-k8scludes1 -n kube-system -- kube-apiserver -h | grep admission-plugins
      --admission-control strings              Admission is divided into two phases. In the first phase, only mutating admission plugins run. In the second phase, only validating admission plugins run. The names in the below list may represent a validating plugin, a mutating plugin, or both. The order of plugins in which they are passed to this flag does not matter. Comma-delimited list of: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. (DEPRECATED: Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.)
      --disable-admission-plugins strings      admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
      --enable-admission-plugins strings       admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

5.2 ResourceQuota资源配额示例

创建目录存放yaml文件。

root@k8scludes1:~# mkdir admissioncontr

root@k8scludes1:~# cd admissioncontr/

创建命名空间。

root@k8scludes1:~/admissioncontr# kubectl create ns admissioncontr
namespace/admissioncontr created

切换命名空间到admissioncontr。

root@k8scludes1:~/admissioncontr# kubens admissioncontr
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "admissioncontr".

编辑pod配置文件，使用nginx镜像生成pod。

root@k8scludes1:~/admissioncontr# vim pod.yaml 

root@k8scludes1:~/admissioncontr# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
pod/podtest created

使用相同的yaml文件，生成另外两个pod。

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
pod/podtest2 created

现在没有任何限制，可以正常创建多个pod。

root@k8scludes1:~/admissioncontr# kubectl get pod -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest    1/1     Running   0          61s   10.244.218.179   k8scludes2   <none>           <none>
podtest1   1/1     Running   0          15s   10.244.218.133   k8scludes2   <none>           <none>
podtest2   1/1     Running   0          9s    10.244.1.84      k8scludes3   <none>           <none>

删除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1 podtest2
pod "podtest" deleted
pod "podtest1" deleted
pod "podtest2" deleted

现在没有设置resourcequota资源配额。

root@k8scludes1:~/admissioncontr# kubectl get quota
No resources found in admissioncontr namespace.

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
No resources found in admissioncontr namespace.

编辑ResourceQuota配置文件，表示创建一个resourcequota资源配额，当前命名空间只能创建2个pod。

resourcequota 资源配额用来设置一个命名空间最多能创建多少个资源对象，比如能创建多少svc，能创建多少pod或者deploy，详细信息请查看博客《Kubernetes(k8s) 资源限制：resources，LimitRange，ResourceQuota》。

root@k8scludes1:~/admissioncontr# vim resourcequota.yaml

root@k8scludes1:~/admissioncontr# cat resourcequota.yaml 
apiVersion: v1 
kind: ResourceQuota 
metadata: 
  name: myresourcequota 
spec: 
  #hard：指定资源的硬性限制，即最大允许使用的资源数量。
  hard: 
    pods: "2"

创建ResourceQuota。

root@k8scludes1:~/admissioncontr# kubectl apply -f resourcequota.yaml 
resourcequota/myresourcequota created

查看resourcequota资源配额，可以看到pod资源限额是2，现在有0个pod。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE   REQUEST     LIMIT
myresourcequota   11s   pods: 0/2   

再次创建pod。

root@k8scludes1:~/admissioncontr#  kubectl apply -f pod.yaml
pod/podtest created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

创建第三个pod的时候报错了，pod资源配额是2，超过2个pod就创建失败了。

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
Error from server (Forbidden): error when creating "STDIN": pods "podtest2" is forbidden: exceeded quota: myresourcequota, requested: pods=1, used: pods=2, limited: pods=2

查看resourcequota，可以看到pods: 2/2，资源配额满了。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE     REQUEST     LIMIT
myresourcequota   5m48s   pods: 2/2   

删除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1
pod "podtest" deleted
pod "podtest1" deleted

5.3 禁用ResourceQuota

resourcequota 资源配额是一种准入控制器插件，默认是启用的。我们可以把resourcequota禁用了。

修改kube-apiserver.yaml文件，禁用resourcequota 资源配额。

disable-admission-plugins=ResourceQuota表示禁用准入控制器插件ResourceQuota。

root@k8scludes1:~/admissioncontr# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:~/admissioncontr# grep disable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml
    - --disable-admission-plugins=ResourceQuota

重启kubelet使配置生效。

root@k8scludes1:~/admissioncontr# systemctl restart kubelet

现在resourcequota 资源配额还在，还是要求当前命名空间只能创建2个pod。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE   REQUEST     LIMIT
myresourcequota   16m   pods: 0/2   

一口气创建了5个pod还是没有报错。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
pod/podtest created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
pod/podtest2 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest3/' pod.yaml | kubectl apply -f -
pod/podtest3 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest4/' pod.yaml | kubectl apply -f -
pod/podtest4 created

root@k8scludes1:~/admissioncontr# kubectl get pod 
NAME       READY   STATUS    RESTARTS   AGE
podtest    1/1     Running   0          30s
podtest1   1/1     Running   0          21s
podtest2   1/1     Running   0          16s
podtest3   1/1     Running   0          11s
podtest4   1/1     Running   0          6s

可以发现，禁用resourcequota功能之后，就算有resourcequota对象存在，资源配额也不生效。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE     REQUEST     LIMIT
myresourcequota   3m59s   pods: 5/2   

删除pod。

root@k8scludes1:~# kubectl delete pod podtest podtest1 podtest2 podtest3 podtest4
pod "podtest" deleted
pod "podtest1" deleted
pod "podtest2" deleted
pod "podtest3" deleted
pod "podtest4" deleted

root@k8scludes1:~# kubectl get pod 
No resources found in admissioncontr namespace.

编辑kube-apiserver.yaml，去掉disable-admission-plugins=ResourceQuota，让ResourceQuota默认启用。

root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:~# grep disable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml 
    #- --disable-admission-plugins=ResourceQuota

重启kubelet使配置生效。

root@k8scludes1:~# systemctl restart kubelet

现在resourcequota 资源配额还是要求当前命名空间只能创建2个pod。

root@k8scludes1:~# kubectl get resourcequota
NAME              AGE     REQUEST     LIMIT
myresourcequota   5h31m   pods: 0/2   

现在ResourceQuota资源配额又生效了，只能创建2个pod。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
Error from server (Forbidden): error when creating "STDIN": pods "podtest2" is forbidden: exceeded quota: myresourcequota, requested: pods=1, used: pods=2, limited: pods=2

删除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1 
pod "podtest" deleted
pod "podtest1" deleted

删除ResourceQuota资源配额。

root@k8scludes1:~/admissioncontr# kubectl delete resourcequota myresourcequota 
resourcequota "myresourcequota" deleted

root@k8scludes1:~/admissioncontr# kubectl get resourcequotas 
No resources found in admissioncontr namespace.

六.配置ImagePolicyWebhook准入控制器禁止使用后缀为latest的镜像

6.1 搭建Webhook服务器

ImagePolicyWebhook的类别为验证。ImagePolicyWebhook 准入控制器允许使用后端 Webhook 做出准入决策。此准入控制器默认被禁用。

准入 Webhook 是一种用于接收准入请求并对其进行处理的 HTTP 回调机制。 可以定义两种类型的准入 webhook，即验证性质的准入 Webhook 和 修改性质的准入 Webhook。 修改性质的准入 Webhook 会先被调用。它们可以更改发送到 API 服务器的对象，以执行自定义的设置默认值操作。在完成了所有对象修改并且 API 服务器也验证了所传入的对象之后， 验证性质的 Webhook 会被调用，并通过拒绝请求的方式来强制实施自定义的策略。

说明： 如果准入 Webhook 需要保证它们所看到的是对象的最终状态以实施某种策略。 则应使用验证性质的准入 Webhook，因为对象被修改性质 Webhook 看到之后仍然可能被修改。

首先需要找一台机器搭建Webhook服务器，我们使用etcd2机器当Webhook服务器。

首先需要安装docker。

[root@etcd2 ~]# yum -y install docker-ce

查看docker版本。

[root@etcd2 ~]# docker -v
Docker version 20.10.12, build e91ed57

配置docker镜像加速器。

[root@etcd2 ~]# cat /etc/docker/daemon.json
{
    "registry-mirrors": [
        "https://frz7i079.mirror.aliyuncs.com"
    ]
}

拉取flavio/kube-image-bouncer镜像，flavio/kube-image-bouncer这个镜像的功能为：不允许使用后缀为latest的镜像。

[root@etcd2 ~]# docker pull flavio/kube-image-bouncer
Using default tag: latest
latest: Pulling from flavio/kube-image-bouncer
ff3a5c916c92: Pull complete 
4947cf5a98cf: Pull complete 
37ff781c0e4d: Pull complete 
1545c1d26daf: Pull complete 
dd581c9cf10b: Pull complete 
Digest: sha256:01e1e0873d20cee1ffefb45421c798cb26250b7a9384978d34ffb1f57403d501
Status: Downloaded newer image for flavio/kube-image-bouncer:latest
docker.io/flavio/kube-image-bouncer:latest

查看flavio/kube-image-bouncer镜像历史信息，可以发现开放端口为1323(EXPOSE 1323)，使用的是web账号登录(USER web)。关于镜像Dockerfile的详细信息，请查看博客《构建自定义镜像并优化dockerfile文件》。

[root@etcd2 ~]# docker history flavio/kube-image-bouncer
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
3974e07cc0c8   3 years ago   /bin/sh -c #(nop)  EXPOSE 1323                  0B        
      3 years ago   /bin/sh -c #(nop)  ENTRYPOINT ["./kube-image…   0B        
      3 years ago   /bin/sh -c #(nop)  USER web                     0B        
      3 years ago   /bin/sh -c chown -R web:web *                   19.9MB    
      3 years ago   /bin/sh -c #(nop) COPY file:de99d16a3731b75b…   19.9MB    
      3 years ago   /bin/sh -c adduser -h /app -D web               4.79kB    
      3 years ago   /bin/sh -c #(nop) WORKDIR /app                  0B        
      4 years ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B        
      4 years ago   /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb…   4.15MB    

使用flavio/kube-image-bouncer镜像创建容器。

-vpwd/webhook-key.pem:/certs/webhook-key.pem:ro -v pwd/webhook.pem:/certs/webhook.pem:ro 表示做目录映射（-v 本地目录：容器目录），webhook-key.pem公钥和webhook.pem私钥不存在也没问题，ro表示容器只有只读权限。

-p 1323:1323表示做端口映射（物理机端口1323：容器端口1323）。关于容器更多详细信息，请查看博客《一文搞懂docker容器基础：docker镜像管理，docker容器管理》。

[root@etcd2 ~]# docker run -dit --name=c1 --restart=always -v `pwd`/webhook-key.pem:/certs/webhook-key.pem:ro -v `pwd`/webhook.pem:/certs/webhook.pem:ro -p 1323:1323 flavio/kube-image-bouncer
7c677fb67522073932617f07ad72cd1a17e60eddc6a68dd25ffe0b8c4f80aaae

容器创建成功了，这时候访问192.168.110.131:1323 就访问到该webhook服务器了。

[root@etcd2 ~]# docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED          STATUS         PORTS                                       NAMES
7c677fb67522   flavio/kube-image-bouncer     "./kube-image-bouncer"   10 seconds ago   Up 8 seconds   0.0.0.0:1323->1323/tcp, :::1323->1323/tcp   c1

6.2 配置kubernetes连接后端webhook服务器

回到kubernetes集群。

root@k8scludes1:~/admissioncontr# cd /etc/kubernetes/

root@k8scludes1:/etc/kubernetes# ls
admin.conf  admission-control-config-file  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf

root@k8scludes1:/etc/kubernetes# cd admission-control-config-file/

root@k8scludes1:/etc/kubernetes/admission-control-config-file# ls
admission_configuration.json  apiserver-client-key.pem  apiserver-client.pem  kubeconfig.yaml  webhook-key.pem  webhook.pem

root@k8scludes1:/etc/kubernetes/admission-control-config-file# pwd
/etc/kubernetes/admission-control-config-file

ImagePolicyWebhook 使用配置文件来为后端行为设置选项。该文件可以是 JSON 或 YAML。

admission_configuration.json是ImagePolicyWebhook 的配置文件，参数解释如下：

• allowTTL以秒计的时长，控制批准请求的缓存时间；
• denyTTL以秒计的时长，控制拒绝请求的缓存时间；
• retryBackoff以毫秒计的时长，控制重试间隔；
• defaultAllow确定 Webhook 后端失效时的行为；
• kubeConfigFile指定kubeconfig文件的路径。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim admission_configuration.json 

root@k8scludes1:/etc/kubernetes/admission-control-config-file# cat admission_configuration.json 
{
  "imagePolicy": {
     "kubeConfigFile": "/etc/kubernetes/admission-control-config-file/kubeconfig.yaml",
     "allowTTL": 50,
     "denyTTL": 50,
     "retryBackoff": 500,
     "defaultAllow": true
  }
}

root@k8scludes1:/etc/kubernetes/admission-control-config-file# ls /etc/kubernetes/admission-control-config-file/admission_configuration.json 
/etc/kubernetes/admission-control-config-file/admission_configuration.json

kubeconfig.yaml用来设置与后端Webhook服务器的连接，要求后端使用 TLS 进行通信。

kubeconfig.yaml文件的 clusters 字段需要指向远端Webhook服务，server指定webhook服务器进行连接。

users 字段需要包含已返回的授权者。关于kubeconfig文件的详细信息，请查看博客《Kubernetes(k8s)访问控制：身份认证》。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim kubeconfig.yaml 

#指定各种证书
root@k8scludes1:/etc/kubernetes/admission-control-config-file# cat kubeconfig.yaml 
apiVersion: v1
kind: Config
clusters:
- cluster:
    # CA 用于验证远程服务
    certificate-authority: /etc/kubernetes/admission-control-config-file/webhook.pem
    # 要查询的远程服务的 URL
    server: http://192.168.110.131:1323/image_policy
  name: bouncer_webhook
contexts:
- context:
    cluster: bouncer_webhook
    user: api-server
  name: bouncer_validator
current-context: bouncer_validator
preferences: {}
users:
- name: api-server
  user:
    # Webhook 准入控制器使用的证书
    client-certificate: /etc/kubernetes/admission-control-config-file/apiserver-client.pem
    # 证书匹配的密钥
    client-key:  /etc/kubernetes/admission-control-config-file/apiserver-client-key.pem

ImagePolicyWebhook默认是没有开启的，现在启用ImagePolicyWebhook。

enable-admission-plugins=NodeRestriction,ImagePolicyWebhook表示启用ImagePolicyWebhook准入控制器。

通过 --admission-control-config-file 参数指定准入控制器ImagePolicyWebhook配置文件的位置。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:/etc/kubernetes/admission-control-config-file# grep admission /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
    - --admission-control-config-file=/etc/kubernetes/admission-control-config-file/admission_configuration.json

重启kubelet使配置生效。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# systemctl restart kubelet

可以发现kube-apiserver.yaml添加参数之后，连接不上集群了。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# kubectl get node
The connection to the server 192.168.110.128:6443 was refused - did you specify the right host or port?

下面开始排查问题，kubectl使用不了，那就只能使用docker排查问题了，使用docker查看容器，可以发现k8s_kube-apiserver容器处于退出状态。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# cd

root@k8scludes1:~# docker ps -a | grep api
2aacfa2bdbc4   e64579b7d886                                        "kube-apiserver --ad…"   19 seconds ago      Exited (1) 18 seconds ago             k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_054ae2c0df7fd5edd4ebb3b7057a0462_8
cb0e2e181dda   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 4 minutes ago       Up 4 minutes                          k8s_POD_kube-apiserver-k8scludes1_kube-system_054ae2c0df7fd5edd4ebb3b7057a0462_0
ac234a8ee92b   e64579b7d886                                        "kube-apiserver --ad…"   4 minutes ago       Exited (1) 4 minutes ago              k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_f38197908cec7cb32a42e0862e367c1c_0
49e9b9c1b389   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 4 minutes ago       Up 4 minutes                          k8s_POD_kube-apiserver-k8scludes1_kube-system_f38197908cec7cb32a42e0862e367c1c_0
5d4985a8833c   e64579b7d886                                        "kube-apiserver --ad…"   5 minutes ago       Exited (1) 5 minutes ago              k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_3dc64835bcbf5bdf937660d07f41b90b_87
c674a6a6c794   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 About an hour ago   Up About an hour                      k8s_POD_kube-apiserver-k8scludes1_kube-system_3dc64835bcbf5bdf937660d07f41b90b_2

查看k8s_kube-apiserver容器日志，报错为：“open /etc/kubernetes/admission-control-config-file/admission_configuration.json: no such file or directory“，发现admission_configuration.json文件找不到。

root@k8scludes1:~# docker logs 2aacfa2bdbc4
I0616 01:39:59.927352       1 server.go:553] external host was not specified, using 192.168.110.128
I0616 01:39:59.927965       1 server.go:161] Version: v1.22.2
Error: failed to initialize admission: failed to read plugin config: unable to read admission control configuration from "/etc/kubernetes/admission-control-config-file/admission_configuration.json" [open /etc/kubernetes/admission-control-config-file/admission_configuration.json: no such file or directory]

/etc/kubernetes/admission-control-config-file/admission_configuration.json文件是存在的，但是报错。

原因为：/etc/kubernetes/admission-control-config-file/admission_configuration.json这个文件是在宿主机里的，并不在容器里。

解决方法：做一个数据卷，把宿主机里的文件映射到容器里。

root@k8scludes1:~# ls /etc/kubernetes/admission-control-config-file/admission_configuration.json
/etc/kubernetes/admission-control-config-file/admission_configuration.json

注意：生产环境里可以先备份下kube-apiserver.yaml 文件再修改，以免改不回来。

定义一个数据卷把宿主机的/etc/kubernetes/admission-control-config-file目录挂载到容器/etc/kubernetes/admission-control-config-file目录。

root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:~# grep -A3 admission-control-config-file /etc/kubernetes/manifests/kube-apiserver.yaml
    - --admission-control-config-file=/etc/kubernetes/admission-control-config-file/admission_configuration.json
    #- --disable-admission-plugins=ResourceQuota
    #- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
    - --token-auth-file=/etc/kubernetes/pki/mytok.csv
--
    - mountPath: /etc/kubernetes/admission-control-config-file
      name: admissionconfile
      readOnly: true
  hostNetwork: true
--
      path: /etc/kubernetes/admission-control-config-file
      type: DirectoryOrCreate
    name: admissionconfile
  - hostPath:

直接上截图可能更直观：

重启kubelet使配置生效。

root@k8scludes1:~# systemctl restart kubelet 

现在k8s正常了。

root@k8scludes1:~# kubectl get pod 
No resources found in admissioncontr namespace.

6.3 验证

编辑pod配置文件，表示使用hub.c.163.com/library/nginx:latest镜像创建pod。

root@k8scludes1:~# cd admissioncontr/

root@k8scludes1:~/admissioncontr# vim pod.yaml 

root@k8scludes1:~/admissioncontr# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod，可以发现：使用tag为latest的镜像不能创建pod。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
Error from server (Forbidden): error when creating "pod.yaml": pods "podtest" is forbidden: image policy webhook backend denied one or more images: Images using latest tag are not allowed

修改pod配置文件，表示使用hub.c.163.com/library/nginx:test镜像创建pod。

root@k8scludes1:~/admissioncontr# vim pod.yaml 

root@k8scludes1:~/admissioncontr# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:test
  #- image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

镜像tag为test，pod创建成功。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
pod/podtest created

root@k8scludes1:~/admissioncontr# kubectl get pod -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          6s    10.244.218.145   k8scludes2              

删除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest 
pod "podtest" deleted

七.总结

准入控制器是Kubernetes中一个非常重要的组件，它负责拦截和处理来自用户或其他应用程序的API请求。通过使用准入控制器，我们可以实现对集群资源的细粒度控制，从而提高集群的安全性和可靠性。