Pod安全策略：PodSecurityPolicy(PSP)

目录

• 一.系统环境
• 二.前言
• 三.PodSecurityPolicy 简介
• 四.为什么需要 PodSecurityPolicy
• 五.给客户端授权
• 六.启用PodSecurityPolicy（PSP）
• 七.PSP规则之禁止创建特权用户pod
• 八.PSP规则之数据卷
  • 8.1 限定数据卷使用某一个目录
  • 8.2 限定数据卷的类型为emptyDir
• 九.PSP规则之指定使用宿主机网络
• 十.PSP规则之pod使用特定的UID运行
• 十一.PSP规则之指定hostport端口
• 十二.PSP规则实战1
• 十三.PSP规则实战2:创建deployment
• 十四.总结

一.系统环境

本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。

服务器版本	docker软件版本	Kubernetes(k8s)集群版本	CPU架构
Ubuntu 18.04.5 LTS	Docker version 20.10.14	v1.22.2	x86_64

Kubernetes集群架构：k8scludes1作为master节点，k8scludes2，k8scludes3作为worker节点。

服务器	操作系统版本	CPU架构	进程	功能描述
k8scludes1/192.168.110.128	Ubuntu 18.04.5 LTS	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master节点
k8scludes2/192.168.110.129	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker节点
k8scludes3/192.168.110.130	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker节点

二.前言

随着容器化和微服务架构的普及，Kubernetes 已经成为企业中编排和管理容器的主要平台。然而，随着集群中运行的容器数量和种类的增加，确保容器的安全性变得越来越重要。PodSecurityPolicy（PSP）是 Kubernetes 用于限制 Pod 创建的一组规则，以确保集群的安全性。在本文中，我们将深入了解 PSP 的概念、重要性以及如何在 Kubernetes 集群中实施。

使用PodSecurityPolicy（PSP）的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。

三.PodSecurityPolicy 简介

PodSecurityPolicy 是一个内置的准入控制器， 允许集群管理员控制 Pod 规约中涉及安全的敏感内容。

首先，在集群中创建一个或多个 PodSecurityPolicy 资源来定义 Pod 必须满足的要求。 然后，创建 RBAC 规则来决定为特定的 Pod 应用哪个 PodSecurityPolicy。 如果 Pod 满足其 PSP 的要求，则照常被允许进入集群。 在某些情况下，PSP 还可以修改 Pod 字段，有效地为这些字段创建新的默认值。 如果 Pod 不符合 PSP 要求，则被拒绝进入集群，并且无法运行。

简单说来，就是PodSecurityPolicy 可以设置pod里的进程只能以某个UID运行，或者设置pod只能使用某一类型存储，不能使用其他，比如只能使用nfs存储。

四.为什么需要 PodSecurityPolicy

在 Kubernetes 中，我们定义了 Deployment、StatefulSet 和 Service 等资源。 这些资源代表软件应用程序的构建块。Kubernetes 集群中的各种控制器根据这些资源做出反应， 创建更多的 Kubernetes 资源或配置一些软件或硬件来实现我们的目标。

在大多数 Kubernetes 集群中，由 RBAC（基于角色的访问控制）规则 控制对这些资源的访问。 list、get、create、edit 和 delete 是 RBAC 关心的 API 操作类型， 但 RBAC 不考虑其所控制的资源中加入了哪些设置。例如， Pod 几乎可以是任何东西，例如简单的网络服务器，或者是特权命令提示（提供对底层服务器节点和所有数据的完全访问权限）。 这对 RBAC 来说都是一样的：Pod 就是 Pod 而已。

要控制集群中定义的资源允许哪些类型的设置，除了 RBAC 之外，还需要准入控制。 从 Kubernetes 1.3 开始，内置 PodSecurityPolicy 一直被作为 Pod 安全相关字段的准入控制机制。 使用 PodSecurityPolicy，可以防止“创建 Pod”这个能力自动变成“每个集群节点上的 root 用户”， 并且无需部署额外的外部准入控制器。

五.给客户端授权

创建minsvcbug目录存放配置文件。

root@k8scludes1:~# mkdir minsvcbug

root@k8scludes1:~# cd minsvcbug

root@k8scludes1:~/minsvcbug# pwd
/root/minsvcbug

创建minsvcbug命名空间。

root@k8scludes1:~/minsvcbug# kubectl create ns minsvcbug
namespace/minsvcbug created

切换命名空间。

root@k8scludes1:~/minsvcbug# kubens minsvcbug
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "minsvcbug".

root@k8scludes1:~/minsvcbug# kubectl get pod 
No resources found in minsvcbug namespace.

本次使用etcd2机器作为k8s客户端，kctom为kubeconfig文件，里面的用户为tom,需要给tom授权，客户端才能具有相关访问权限。

关于kubeconfig文件的详细操作，请查看博客《Kubernetes(k8s)访问控制：身份认证》。

关于授权的详细操作，请查看博客《Kubernetes(k8s)访问控制：权限管理之RBAC鉴权》。

[root@etcd2 ~]# ls
kctom  

查看minsvcbug命名空间下的pod，现在tom用户还没有minsvcbug命名空间的pod访问权限。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n minsvcbug
Error from server (Forbidden): pods is forbidden: User "tom" cannot list resource "pods" in API group "" in the namespace "minsvcbug"

给客户端tom用户授予创建/删除/查看pod的权限，编辑角色role配置文件。

root@k8scludes1:~/minsvcbug# vim role1.yaml 

#role1角色对pod,svc,secret具有get,list,watch,create,delete权限
root@k8scludes1:~/minsvcbug# cat role1.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: role1
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - secrets
  verbs:
  - get
  - list
  - watch
  - create
  - delete

创建角色role1。

root@k8scludes1:~/minsvcbug# kubectl apply -f role1.yaml 
role.rbac.authorization.k8s.io/role1 created

查看角色。

root@k8scludes1:~/minsvcbug# kubectl get role -o wide
NAME    CREATED AT
role1   2022-05-20T03:42:48Z

查看role1的详细描述。

root@k8scludes1:~/minsvcbug# kubectl describe role role1 
Name:         role1
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods       []                 []              [get list watch create delete]
  secrets    []                 []              [get list watch create delete]
  services   []                 []              [get list watch create delete]

把角色role1绑定给tom用户。

root@k8scludes1:~/minsvcbug# kubectl create rolebinding rolebindingtotom --role=role1 --user=tom
rolebinding.rbac.authorization.k8s.io/rolebindingtotom created

root@k8scludes1:~/minsvcbug# kubectl get rolebinding
NAME               ROLE         AGE
rolebindingtotom   Role/role1   11s

给tom用户授予创建/删除/查看pod的权限之后，在客户端查看。可以看到tom用户对minsvcbug命名空间的pod有访问权限了。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

可以看到tom用户对default命名空间的pod没有访问权限。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom 
Error from server (Forbidden): pods is forbidden: User "tom" cannot list resource "pods" in API group "" in the namespace "default"

在客户端etcd2机器上创建目录存放文件。

[root@etcd2 ~]# mkdir minsvcbug

[root@etcd2 ~]# cd minsvcbug/

[root@etcd2 minsvcbug]# pwd
/root/minsvcbug

编辑pod配置文件，使用nginx创建pod。

[root@etcd2 minsvcbug]# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

在客户端使用nginx镜像创建pod。

[root@etcd2 minsvcbug]# kubectl apply -f pod.yaml --kubeconfig=kctom -n minsvcbug
pod/podtest created

pod创建成功，说明成功授权了。

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          36s   10.244.218.132   k8scludes2   <none>           <none>

[root@etcd2 minsvcbug]# kubectl delete -f pod.yaml --kubeconfig=kctom -n minsvcbug
pod "podtest" deleted

现在tom用户就具备了minsvcbug命名空间的创建/删除/查看pod的权限了。

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

六.启用PodSecurityPolicy（PSP）

接下来启用PSP，编辑kube-apiserver.yaml文件。

root@k8scludes1:~/minsvcbug# ls /etc/kubernetes/manifests/kube-apiserver.yaml 
/etc/kubernetes/manifests/kube-apiserver.yaml

启用PSP需要添加- --enable-admission-plugins=PodSecurityPolicy。

root@k8scludes1:~/minsvcbug# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:~/minsvcbug# grep enable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

重启kubelet使配置生效。

root@k8scludes1:~/minsvcbug# systemctl restart kubelet

root@k8scludes1:~/minsvcbug# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   33d   v1.22.2
k8scludes2   Ready    <none>                 33d   v1.22.2
k8scludes3   Ready    <none>                 33d   v1.22.2

PSP(pod security policy)默认没有开启，如果开启PSP之后，没有写PSP规则的话，任何用户都创建不了POD，包括管理员。

注意：一定要等k8s正常工作之后再启动PSP，如果etcd,api都没启动好就启动PSP，则相关pod都启动不了了 ，等PSP启动之后再去写PSP规则。

启用PSP之后，创建pod必须先访问PSP规则，通过了PSP规则才能创建POD，否则创建pod失败。

默认情况下，普通用户没有访问PSP规则的权限，管理员用户可以直接访问PSP规则，创建好PSP规则之后，对普通用户要做一个RBAC的权限设置，使其可以访问某个PSP规则，再决定是不是能够把pod创建出 来(1.是否有足够的RBAC权限 2.创建的POD是否符合PSP规则)。

七.PSP规则之禁止创建特权用户pod

查看PSP规则。

root@k8scludes1:~/minsvcbug# kubectl get psp
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME         PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller   false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
speaker      true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

如果没有任何PSP规则，则任何用户都不能创建pod。

编辑PSP规则，我们的目的是拒绝特权pod运行，privileged: false 表示不允许特权pod运行。

root@k8scludes1:~/minsvcbug# vim psp-deny-privileges-pod.yaml

root@k8scludes1:~/minsvcbug# cat psp-deny-privileges-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-deny-privileges-pod
spec: 
  # Don't allow privileged pods!
  privileged: false 
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
  fsGroup: 
    rule: RunAsAny 
  volumes: 
  - '*'

创建psp。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-deny-privileges-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-deny-privileges-pod created

psp-deny-privileges-pod规则就创建好了。

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME                      PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller                false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-deny-privileges-pod   false             RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker                   true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

创建psp-deny-privileges-pod的PSP规则之后,在客户端使用普通用户创建pod。

编辑pod配置文件。

[root@etcd2 minsvcbug]# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod失败，就算普通用户tom创建没有特权用户的pod，因为tom没有访问PSP规则的权限，所以创建pod失败。

[root@etcd2 minsvcbug]# kubectl apply -f pod.yaml --kubeconfig=kctom -n minsvcbug
Error from server (Forbidden): error when creating "pod.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: []

[root@etcd2 minsvcbug]# kubectl get pod --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

下面要给tom用户授予PSP的访问权限，对PSP的授权可以是clusterrole，也可以是role，PSP规则是全局生效的，在其他命名空间也可以看到，本次使用clusterrole授权。

生成clusterrole-access-psp-rule的yaml文件，表示：角色clusterrole-access-psp-rule对名为psp-deny-privileges-pod有use权限。

root@k8scludes1:~# kubectl create clusterrole clusterrole-access-psp-rule --verb=use --resource=psp --resource-name=psp-deny-privileges-pod --dry-run=client -o yaml >clusterrole-access-psp-rule.yaml

生成的yaml文件如下：

root@k8scludes1:~# vim clusterrole-access-psp-rule.yaml 

root@k8scludes1:~# cat clusterrole-access-psp-rule.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: clusterrole-access-psp-rule
rules:
- apiGroups:
  - policy
  resourceNames:
  - psp-deny-privileges-pod
  resources:
  - podsecuritypolicies
  verbs:
  - use

创建clusterrole-access-psp-rule。

root@k8scludes1:~# kubectl apply -f clusterrole-access-psp-rule.yaml 
clusterrole.rbac.authorization.k8s.io/clusterrole-access-psp-rule created

root@k8scludes1:~# kubectl get clusterrole | grep clusterrole-access-psp-rule
clusterrole-access-psp-rule                                            2022-05-20T10:21:15Z

查看角色的详细描述。

root@k8scludes1:~# kubectl describe clusterrole clusterrole-access-psp-rule 
Name:         clusterrole-access-psp-rule
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources                   Non-Resource URLs  Resource Names             Verbs
  ---------                   -----------------  --------------             -----
  podsecuritypolicies.policy  []                 [psp-deny-privileges-pod]  [use]

把角色clusterrole-access-psp-rule绑定给tom用户。

root@k8scludes1:~# kubectl create rolebinding rolebinding-clusterrole-access-psp --clusterrole=clusterrole-access-psp-rule --user=tom
rolebinding.rbac.authorization.k8s.io/rolebinding-clusterrole-access-psp created

root@k8scludes1:~# kubectl get rolebinding -o wide
NAME                                 ROLE                                      AGE     USERS   GROUPS   SERVICEACCOUNTS
rolebinding-clusterrole-access-psp   ClusterRole/clusterrole-access-psp-rule   15s     tom              
rolebindingtotom                     Role/role1                                6h40m   tom        

给tom用户授予PSP的访问权限之后，到客户端创建无特权用户的pod。

[root@etcd2 minsvcbug]# kubectl apply -f pod.yaml --kubeconfig=kctom -n minsvcbug
pod/podtest created

pod创建成功，现在tom用户就可以创建无特权用户的pod了。

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          21s   10.244.218.145   k8scludes2   <none>           <none>

删除pod。

[root@etcd2 minsvcbug]# kubectl delete pod podtest --kubeconfig=kctom -n minsvcbug
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

编辑pod文件，securityContext:privileged: true 表示创建特权用户的pod。

[root@etcd2 minsvcbug]# cat podprivileged.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    #创建特权pod
    securityContext:
      privileged: true
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod失败，Privileged containers are not allowed 创建特权容器是不允许的。

[root@etcd2 minsvcbug]# kubectl apply -f podprivileged.yaml --kubeconfig=kctom -n minsvcbug
Error from server (Forbidden): error when creating "podprivileged.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

删除psp-deny-privileges-pod规则。

root@k8scludes1:~/minsvcbug# kubectl delete psp psp-deny-privileges-pod
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy "psp-deny-privileges-pod" deleted

root@k8scludes1:~/minsvcbug# kubectl get psp
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME         PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller   false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
speaker      true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

删除clusterrole-access-psp-rule(不删除也可以)。

root@k8scludes1:~/minsvcbug# kubectl get clusterrole | grep clusterrole-access-psp-rule
clusterrole-access-psp-rule                                            2022-05-20T10:21:15Z

root@k8scludes1:~/minsvcbug# kubectl delete clusterrole clusterrole-access-psp-rule
clusterrole.rbac.authorization.k8s.io "clusterrole-access-psp-rule" deleted

删除rolebinding-clusterrole-access-psp(不删除也可以)。

root@k8scludes1:~/minsvcbug# kubectl delete rolebinding rolebinding-clusterrole-access-psp
rolebinding.rbac.authorization.k8s.io "rolebinding-clusterrole-access-psp" deleted

总结：

• 在启用了PSP之后，没有任何PSP规则的情况下，任何用户(包括管理员)都不能创建pod；
• 在启用了PSP之后，且创建了PSP规则，要确保用户能访问到此PSP规则，创建clusterrole，让用户能use某条PSP规则；
• 能不能成功创建pod，决定权在于是否具备创建pod的RBAC权限，有对应的RBAC权限之后，创建的pod是否满足PSP规则。

八.PSP规则之数据卷

8.1 限定数据卷使用某一个目录

编辑psp配置文件，privileged: true表示可以创建特权用户的pod，volumes:- '*' 表示可以使用所有类型的数据卷，allowedHostPaths:- pathPrefix: "/tmp" 表示数据卷只能挂载/tmp目录，psp-mount-pod这条PSP规则规定hostPath只能使用/tmp目录。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
  fsGroup: 
    rule: RunAsAny 
  volumes: 
  - '*'
  allowedHostPaths:
    - pathPrefix: "/tmp"

创建PSP。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod created

root@k8scludes1:~/minsvcbug# kubectl get psp
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

创建了新的PSP规则之后，需要重新给客户端的tom用户授权。

创建clusterrole--psp-mount-rule，角色clusterrole--psp-mount-rule对名为psp-mount-pod的psp规则有use权限。

root@k8scludes1:~/minsvcbug# kubectl create clusterrole clusterrole--psp-mount-rule --verb=use --resource=psp --resource-name=psp-mount-pod
clusterrole.rbac.authorization.k8s.io/clusterrole--psp-mount-rule created

把角色clusterrole--psp-mount-rule绑定给tom用户。

root@k8scludes1:~/minsvcbug# kubectl create rolebinding rolebinding-clusterrole-mount-psp --clusterrole=clusterrole--psp-mount-rule --user=tom
rolebinding.rbac.authorization.k8s.io/rolebinding-clusterrole-mount-psp created

给客户端tom用户授权之后，编辑pod配置文件，volumes:- name: v1 hostPath: path: /opt 表示创建名叫v1的数据卷，目录为/opt ,把v1这个数据卷挂载到容器的/data目录。securityContext:privileged: true表示创建特权pod。

关于数据卷的详细操作，请查看博客《Kubernetes(k8s)存储管理之数据卷volumes(一)：volumes的引入和emptyDir数据卷》。

[root@etcd2 minsvcbug]# vim podmount.yaml 

[root@etcd2 minsvcbug]# cat podmount.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  #数据卷v1
  volumes:
  - name: v1
    hostPath:
      path: /opt
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    #把数据卷v1挂载到/data目录
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod失败，"/opt": is not allowed to be used] ，/opt目录不被允许使用。

[root@etcd2 minsvcbug]# kubectl apply -f podmount.yaml --kubeconfig=kctom -n minsvcbug
Error from server (Forbidden): error when creating "podmount.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[0].hostPath.pathPrefix: Invalid value: "/opt": is not allowed to be used]

修改pod配置文件，把v1数据卷的/opt目录换成/tmp目录。

[root@etcd2 minsvcbug]# vim podmount.yaml 

[root@etcd2 minsvcbug]# cat podmount.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    hostPath:
      path: /tmp
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod。

[root@etcd2 minsvcbug]# kubectl apply -f podmount.yaml --kubeconfig=kctom -n minsvcbug
pod/podtest created

把v1数据卷的/opt目录换成/tmp目录之后，pod就创建成功了。

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          14s   10.244.218.134   k8scludes2   <none>           <none>

[root@etcd2 minsvcbug]# kubectl delete pod podtest --kubeconfig=kctom -n minsvcbug
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

8.2 限定数据卷的类型为emptyDir

编辑PSP配置文件，volumes:- 'emptyDir' 限定数据卷的类型为emptyDir。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
  fsGroup: 
    rule: RunAsAny 
  volumes: 
  - 'emptyDir'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"

创建PSP。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            emptyDir
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

编辑pod配置文件，hostPath:path: /tmp 指定hostPath目录，这里使用的是hostPath类型的数据卷。

root@k8scludes1:~/minsvcbug# cat podmount.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    hostPath:
      path: /tmp
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod失败，Invalid value: "hostPath": hostPath volumes are not allowed to be used 表示不能使用hostPath数据卷。

root@k8scludes1:~/minsvcbug# kubectl apply -f podmount.yaml 
Error from server (Forbidden): error when creating "podmount.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]

这次在客户端创建pod。

修改pod配置文件，emptyDir: {}表示使用emptyDir数据卷。

[root@etcd2 minsvcbug]# cat podmount.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod失败，pod的数据卷类型修改为emptyDir之后，还是报错：Invalid value: "projected": projected volumes are not allowed to be used]。

[root@etcd2 minsvcbug]# kubectl apply -f podmount.yaml --kubeconfig=kctom -n minsvcbug
Error from server (Forbidden): error when creating "podmount.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]

先修改PSP规则，数据卷类型还是设置为所有类型volumes:- "*" 。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - "*" 
    #- 'emptyDir'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"

应用PSP。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

PSP规则的数据卷类型修改为所有类型之后，创建pod成功。

root@k8scludes1:~/minsvcbug# cat podmount.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

root@k8scludes1:~/minsvcbug# kubectl apply -f podmount.yaml 
pod/podtest created

root@k8scludes1:~/minsvcbug# kubectl get pod -o wide 
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          9s    10.244.218.155   k8scludes2   <none>           <none>

使用yaml文件的格式查看podtest，可以发现有两个数据卷，一个是emptyDir类型的，一个是projected类型的。

root@k8scludes1:~/minsvcbug# kubectl get pod podtest -o yaml
apiVersion: v1
kind: Pod
metadata:
  ......
  volumes:
  - emptyDir: {}
    name: v1
  - name: kube-api-access-8xjp8
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
......

更新软件源。

root@k8scludes1:~/minsvcbug# apt-get update

安装jq,jq可以使JSON格式更好看。

root@k8scludes1:~/minsvcbug# apt-get install jq -y 

root@k8scludes1:~/minsvcbug# which jq
/usr/bin/jq

以json格式的方式查看podtest，jsonpath='{.spec.volumes}'表示只查看spec字段下的volumes，这样显示不直观，使用jq使JSON格式化。

root@k8scludes1:~/minsvcbug# kubectl get pod podtest -o jsonpath='{.spec.volumes}'
[{"emptyDir":{},"name":"v1"},{"name":"kube-api-access-8xjp8","projected":{"defaultMode":420,"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"items":[{"key":"ca.crt","path":"ca.crt"}],"name":"kube-root-ca.crt"}},{"downwardAPI":{"items":[{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"},"path":"namespace"}]}}]}}]root@k8scludes1:~/minsvcbug#

可以发现不仅有emptyDir数据卷，还有projected类型的数据卷。

root@k8scludes1:~/minsvcbug# kubectl get pod podtest -o jsonpath='{.spec.volumes}' | jq
[
  {
    "emptyDir": {},
    "name": "v1"
  },
  {
    "name": "kube-api-access-8xjp8",
    "projected": {
      "defaultMode": 420,
      "sources": [
        {
          "serviceAccountToken": {
            "expirationSeconds": 3607,
            "path": "token"
          }
        },
        {
          "configMap": {
            "items": [
              {
                "key": "ca.crt",
                "path": "ca.crt"
              }
            ],
            "name": "kube-root-ca.crt"
          }
        },
        {
          "downwardAPI": {
            "items": [
              {
                "fieldRef": {
                  "apiVersion": "v1",
                  "fieldPath": "metadata.namespace"
                },
                "path": "namespace"
              }
            ]
          }
        }
      ]
    }
  }
]

所以创建PSP规则的时候不仅要指定emptyDir类型的数据卷，还要指定projected类型的数据卷，修改PSP规则。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - 'emptyDir'
  - 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"

应用PSP规则。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            emptyDir,projected
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

删除pod。

root@k8scludes1:~/minsvcbug# kubectl get pod 
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          13m

root@k8scludes1:~/minsvcbug# kubectl delete pod podtest 
pod "podtest" deleted

修改pod配置文件，数据卷类型为emptyDir。

[root@etcd2 minsvcbug]# cat podmount.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

客户端继续创建pod，在客户端成功创建emptyDir类型数据卷的pod。

[root@etcd2 minsvcbug]# kubectl apply -f podmount.yaml --kubeconfig=kctom -n minsvcbug
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          6s    10.244.218.136   k8scludes2   <none>           <none>

删除pod。

[root@etcd2 minsvcbug]# kubectl delete pod podtest --kubeconfig=kctom -n minsvcbug
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
No resources found in minsvcbug namespace.

九.PSP规则之指定使用宿主机网络

编辑pod配置文件，hostNetwork: true 表示是否使用宿主机网络，true表示使用宿主机网络，默认是false，不使用宿主机网络。

hostNetwork: true相当于docker run -dit --name=nginx --network=net nginx 共享宿主机网络空间。

[root@etcd2 minsvcbug]# vim podnetwork.yaml 

[root@etcd2 minsvcbug]# cat podnetwork.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  #hostNetwork: true 表示pod使用宿主机网络 
  hostNetwork: true
  restartPolicy: Always
status: {}

创建pod失败，Invalid value: true: Host network is not allowed to be used]表示 pod不能使用宿主机网络。

[root@etcd2 minsvcbug]# kubectl apply -f podnetwork.yaml --kubeconfig=kctom -n minsvcbug
Error from server (Forbidden): error when creating "podnetwork.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used]

修改PSP规则，使pod可以使用宿主机网络，hostNetwork: true 表示可以使用宿主机网络。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  hostNetwork: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - 'emptyDir'
  - 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"

应用PSP规则。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            emptyDir,projected
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

客户端再次创建使用宿主机网络的pod，这次pod创建成功。

[root@etcd2 minsvcbug]# kubectl apply -f podnetwork.yaml --kubeconfig=kctom -n minsvcbug
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -o wide --kubeconfig=kctom -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          17s   192.168.110.129   k8scludes2   <none>           <none>

删除pod。

[root@etcd2 minsvcbug]# kubectl delete pod podtest  --kubeconfig=kctom -n minsvcbug
pod "podtest" deleted

在default命名空间，创建宿主机网络的pod。

root@k8scludes1:~/minsvcbug# kubectl apply -f podnetwork.yaml 
pod/podtest created

root@k8scludes1:~/minsvcbug# kubectl get pod -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          7s    192.168.110.129   k8scludes2   <none>           <none>

进入pod。

#可以看到pod里使用的是宿主机里的网络
root@k8scludes1:~/minsvcbug# kubectl exec -it podtest -- bash
root@k8scludes2:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:71:45:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.110.129/24 brd 192.168.110.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe71:45c1/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:fd:bc:7b:4b brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.244.218.128/32 scope global tunl0
       valid_lft forever preferred_lft forever
       
#退出pod       
root@k8scludes2:/# exit
exit

删除pod。

root@k8scludes1:~/minsvcbug# kubectl delete pod podtest 
pod "podtest" deleted

root@k8scludes1:~/minsvcbug# kubectl get pod 
No resources found in minsvcbug namespace.

十.PSP规则之pod使用特定的UID运行

编辑pod配置文件，hostNetwork: true 表示pod使用宿主机网络 。

root@k8scludes1:~/minsvcbug# cat podnetwork.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  hostNetwork: true
  restartPolicy: Always
status: {}

创建pod。

root@k8scludes1:~/minsvcbug# kubectl apply -f podnetwork.yaml 
pod/podtest created

root@k8scludes1:~/minsvcbug# kubectl get pod -o wide 
NAME      READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          12s   192.168.110.129   k8scludes2   <none>           <none>

进入pod，可以发现pod默认是以root身份运行的，可不可以设置pod以其他身份运行呢？

root@k8scludes1:~/minsvcbug# kubectl exec -it podtest -- bash

#查看网络，pod使用的是宿主机的网络
root@k8scludes2:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:71:45:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.110.129/24 brd 192.168.110.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe71:45c1/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:4c:05:bc:65 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.244.218.128/32 scope global tunl0
       valid_lft forever preferred_lft forever

#pod是以root用户运行的
root@k8scludes2:/# id
uid=0(root) gid=0(root) groups=0(root)

#退出pod
root@k8scludes2:/# exit
exit

查看PSP规则。

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            emptyDir,projected
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

编辑PSP规则，runAsUser:rule: RunAsAny 中 runAsUser表示以某种身份运行，RunAsAny表示可以以任何身份运行，rule: MustRunAs ranges:- min: 1000 max: 2000 表示pod必须以1000-2000的UID身份运行。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  hostNetwork: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
  #rule: RunAsAny 
    rule: MustRunAs
    #MustRunAs 表示pod必须以1000-2000的UID身份运行
    ranges:
    - min: 1000
      max: 2000
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - 'emptyDir'
  - 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"

使PSP规则生效。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   MustRunAs   RunAsAny    RunAsAny    false            emptyDir,projected
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

因为客户端etcd2需要进入到pod里，以及查看pod的日志，再给客户端添加pods/exec， pods/log权限，直接在Role配置文件中加入即可。

root@k8scludes1:~/minsvcbug# vim role1.yaml 

root@k8scludes1:~/minsvcbug# cat role1.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: role1
rules:
- apiGroups:
  - ""
  resources:
  - pods
  #pods/exec， pods/log是新加的权限
  - pods/exec
  - pods/log
  - services
  - secrets
  verbs:
  - get
  - list
  - watch
  - create
  - delete

应用Role。

root@k8scludes1:~/minsvcbug# kubectl apply -f role1.yaml 
role.rbac.authorization.k8s.io/role1 configured

查看role1的权限信息。

root@k8scludes1:~/minsvcbug# kubectl describe role role1 
Name:         role1
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods/exec  []                 []              [get list watch create delete]
  pods/log   []                 []              [get list watch create delete]
  pods       []                 []              [get list watch create delete]
  secrets    []                 []              [get list watch create delete]
  services   []                 []              [get list watch create delete]

接下来在客户端etcd2机器进行测试。

把kubeconfig文件kctom复制到~/.kube/config，这样执行命令行就不用每次都指定--kubeconfig了。

[root@etcd2 minsvcbug]# cp kctom ~/.kube/config

[root@etcd2 minsvcbug]# ls ~/.kube/
cache  config

[root@etcd2 minsvcbug]# kubectl get pod -n minsvcbug
No resources found in minsvcbug namespace.

编辑pod文件。

[root@etcd2 minsvcbug]# vim poduid.yaml

[root@etcd2 minsvcbug]# cat poduid.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  #emptyDir类型的数据卷
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    #创建特权pod
    securityContext:
      privileged: true
    resources: {}
    #数据卷挂载
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  #hostNetwork: true 表示pod使用宿主机网络 
  hostNetwork: true
  restartPolicy: Always
status: {}

创建pod。

[root@etcd2 minsvcbug]# kubectl apply -f poduid.yaml -n minsvcbug
pod/podtest created

pod创建失败。

[root@etcd2 minsvcbug]# kubectl get pod -n minsvcbug
NAME      READY   STATUS             RESTARTS     AGE
podtest   0/1     CrashLoopBackOff   1 (5s ago)   6s

查看pod日志：podtest里不允许root来运行，podtest默认运行的是nginx进程，nginx进程需要使用root身份来运行。

[root@etcd2 minsvcbug]# kubectl logs podtest -n minsvcbug
2022/05/24 07:53:04 [warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
2022/05/24 07:53:04 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

删除pod。

[root@etcd2 minsvcbug]# kubectl delete pod podtest -n minsvcbug
pod "podtest" deleted

编辑pod配置文件，这次pod不运行nginx的默认进程，因为nginx进程需要root才能正常运行，我们运行sleep 100000命令，sleep 100000命令什么用户都可以运行。

[root@etcd2 minsvcbug]# vim poduid.yaml

[root@etcd2 minsvcbug]# cat poduid.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    #运行sleep 100000命令
    command: ["sh","-c","sleep 100000"]
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  hostNetwork: true
  restartPolicy: Always
status: {}

创建pod。

[root@etcd2 minsvcbug]# kubectl apply -f poduid.yaml -n minsvcbug
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          12s   192.168.110.129   k8scludes2   <none>           <none>

进入pod。

[root@etcd2 minsvcbug]# kubectl exec -it podtest -n minsvcbug -- bash

#现在默认的用户UID为1000
I have no name!@k8scludes2:/$ id
uid=1000 gid=0(root) groups=0(root)

#退出pod
I have no name!@k8scludes2:/$ exit
exit

删除pod。

[root@etcd2 minsvcbug]# kubectl delete pod podtest -n minsvcbug
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl get pod -n minsvcbug
No resources found in minsvcbug namespace.

编辑pod配置文件，securityContext:runAsUser: 3000 表示pod以UID为3000的身份运行。

[root@etcd2 minsvcbug]# vim poduid.yaml

[root@etcd2 minsvcbug]# cat poduid.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  securityContext:
    runAsUser: 3000
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    command: ["sh","-c","sleep 100000"]
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod失败，UID 3000不在PSP规则规定的{1000 2000}之间，所以创建pod失败。

[root@etcd2 minsvcbug]# kubectl apply -f poduid.yaml -n minsvcbug
Error from server (Forbidden): error when creating "poduid.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.runAsUser: Invalid value: 3000: must be in the ranges: [{1000 2000}]]

编辑pod配置文件，securityContext:runAsUser: 1500 表示pod以UID为1500的身份运行。

[root@etcd2 minsvcbug]# vim poduid.yaml

[root@etcd2 minsvcbug]# cat poduid.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  securityContext:
    runAsUser: 1500
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    command: ["sh","-c","sleep 100000"]
    name: podtest
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod成功。

[root@etcd2 minsvcbug]# kubectl apply -f poduid.yaml -n minsvcbug
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          12s

进入pod。

[root@etcd2 minsvcbug]# kubectl exec -it podtest -n minsvcbug -- bash

#UID 1500在{1000 2000}范围，所以pod运行成功，pod运行进程UID也为1500
I have no name!@podtest:/$ id
uid=1500 gid=0(root) groups=0(root)

#退出pod
I have no name!@podtest:/$ exit
exit

十一.PSP规则之指定hostport端口

修改一下vim设置，设置为粘贴模式之后，粘贴格式看着更规范。

root@k8scludes1:~/minsvcbug# vim /etc/vim/vimrc

root@k8scludes1:~/minsvcbug# tail -1 /etc/vim/vimrc
set paste

编辑PSP规则，因为nginx进程需要root身份才能运行，RUNASUSER 变为RunAsAny，RunAsAny表示可以以任何身份运行。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  #hostNetwork: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
    #rule: MustRunAs
    #ranges:
    #- min: 1000
    #  max: 2000
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - '*'
  #- 'emptyDir'
  #- 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"

应用PSP规则。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

去客户端创建pod。

编辑pod配置文件，containerPort: 80 表示容器端口80，hostPort: 8080表示宿主机端口8080 ，表示把nginx容器的80端口映射为宿主机的8080端口。

这样创建的pod类似于docker端口映射：docker run -p 80 nginx 只写一个端口80的话表示容器端口，docker run -p 80:80 nginx 80:80 表示宿主机端口:容器端口。

[root@etcd2 minsvcbug]# vim podport.yaml 

[root@etcd2 minsvcbug]# cat podport.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
    ports:
    - name: http
      containerPort: 80
      hostPort: 8080
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

创建pod的时候发现tom用户没有pods的patch权限，我们去授权一下。

[root@etcd2 minsvcbug]# kubectl apply -f podport.yaml -n minsvcbug
for: "podport.yaml": pods "podtest" is forbidden: User "tom" cannot patch resource "pods" in API group "" in the namespace "minsvcbug"

修改Role配置文件，添加patch权限。

root@k8scludes1:~/minsvcbug# vim role1.yaml 

root@k8scludes1:~/minsvcbug# cat role1.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: role1
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  - pods/log
  - services
  - secrets
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  #添加patch权限
  - patch

应用角色。

root@k8scludes1:~/minsvcbug# kubectl apply -f role1.yaml 
role.rbac.authorization.k8s.io/role1 configured

客户端继续创建pod。

创建pod失败，Invalid value: 8080: Host port 8080 is not allowed to be used. Allowed ports: [] 表示PSP规则规定的Host port为空，不能使用hostport端口，我们需要修改PSP规则。

[root@etcd2 minsvcbug]# kubectl get pod -n minsvcbug
No resources found in minsvcbug namespace.

[root@etcd2 minsvcbug]# kubectl apply -f podport.yaml -n minsvcbug
Error from server (Forbidden): error when creating "podport.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].hostPort: Invalid value: 8080: Host port 8080 is not allowed to be used. Allowed ports: []]

修改PSP规则，hostPorts:- min: 1000 max: 8090表示hostport的端口范围为1000-8090。

root@k8scludes1:~/minsvcbug# vim psp-mount-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-mount-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-mount-pod
spec: 
  #allow privileged pods!
  privileged: true
  #hostNetwork: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
    #rule: MustRunAs
    #ranges:
    #- min: 1000
    #  max: 2000
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - '*'
  #- 'emptyDir'
  #- 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"
  #指定hostport的端口范围为1000-8090
  hostPorts:
  - min: 1000
    max: 8090

应用PSP规则。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-mount-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-mount-pod configured

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

现在客户端创建pod就成功了。

[root@etcd2 minsvcbug]# kubectl apply -f podport.yaml -n minsvcbug
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n minsvcbug
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          53s   10.244.218.150   k8scludes2   <none>           <none>

访问nginx的方法为：curl k8scludes2的ip:8080，nginx访问成功。

[root@etcd2 minsvcbug]# curl 192.168.110.129:8080
DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
style>
head>
<body>
<h1>Welcome to nginx!h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.orga>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.coma>.p>

<p><em>Thank you for using nginx.em>p>
body>
html>

删除pod。

[root@etcd2 minsvcbug]# kubectl delete pod podtest  -n minsvcbug
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n minsvcbug
No resources found in minsvcbug namespace.

删除rolebinding。

root@k8scludes1:~# cd minsvcbug/

root@k8scludes1:~/minsvcbug# kubectl get rolebinding
NAME                                ROLE                                      AGE
rolebinding-clusterrole-mount-psp   ClusterRole/clusterrole--psp-mount-rule   3d21h
rolebindingtotom                    Role/role1                                5d3h

root@k8scludes1:~/minsvcbug# kubectl delete rolebinding rolebinding-clusterrole-mount-psp
rolebinding.rbac.authorization.k8s.io "rolebinding-clusterrole-mount-psp" deleted

root@k8scludes1:~/minsvcbug# kubectl get rolebinding
NAME               ROLE         AGE
rolebindingtotom   Role/role1   5d3h

root@k8scludes1:~/minsvcbug# kubectl delete rolebinding rolebindingtotom
rolebinding.rbac.authorization.k8s.io "rolebindingtotom" deleted

root@k8scludes1:~/minsvcbug# kubectl get rolebinding
No resources found in minsvcbug namespace.

删除role。

root@k8scludes1:~/minsvcbug# kubectl get role
NAME    CREATED AT
role1   2022-05-20T03:42:48Z

root@k8scludes1:~/minsvcbug# kubectl delete role role1 
role.rbac.authorization.k8s.io "role1" deleted

root@k8scludes1:~/minsvcbug# kubectl get role
No resources found in minsvcbug namespace.

删除PSP规则。

root@k8scludes1:~/minsvcbug# kubectl get psp
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME            PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller      false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-mount-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker         true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

#删除psp-mount-pod规则
root@k8scludes1:~/minsvcbug# kubectl delete psp psp-mount-pod
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy "psp-mount-pod" deleted

十二.PSP规则实战1

我们现在要解决的问题是：有三个命名空间ns1,ns2,ns3，tom用户只有在ns1命名空间里可以创建特权用户pod，其他命名空间都不能创建特权用户pod，需要怎么做？

创建3个命名空间。

root@k8scludes1:~/minsvcbug# kubectl create ns ns1
namespace/ns1 created

root@k8scludes1:~/minsvcbug# kubectl create ns ns2
namespace/ns2 created

root@k8scludes1:~/minsvcbug# kubectl create ns ns3
namespace/ns3 created

创建一个普通的RBAC权限，让客户端tom用户可以在所有的命名空间里创建和删除pod,使用clusterrole和clusterrolebinding。

编辑ClusterRole配置文件，clusterole1具有对pod的查询/创建/删除权限，因为需要看日志和进入到pod里，添加了pods/exec和pods/log资源。

root@k8scludes1:~/minsvcbug# vim clusterrole1.yaml 

root@k8scludes1:~/minsvcbug# cat clusterrole1.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: clusterole1
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  - pods/log
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - patch

应用ClusterRole。

root@k8scludes1:~/minsvcbug# kubectl apply -f clusterrole1.yaml 
clusterrole.rbac.authorization.k8s.io/clusterole1 created

root@k8scludes1:~/minsvcbug# kubectl get clusterrole | grep clusterole1
clusterole1                                                            2022-05-25T07:31:28Z

查看clusterole1权限信息。

root@k8scludes1:~/minsvcbug# kubectl describe clusterrole clusterole1
Name:         clusterole1
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods/exec  []                 []              [get list watch create delete patch]
  pods/log   []                 []              [get list watch create delete patch]
  pods       []                 []              [get list watch create delete patch]

把clusterole1绑定给tom用户，clusterole1和clusterrolebinding1的目的是让tom用户可以在所有命名空间里创建和删除pod。

现在客户端的tom用户在所有命名空间都可以创建和删除pod了。

root@k8scludes1:~/minsvcbug# kubectl create clusterrolebinding clusterrolebinding1 --clusterrole=clusterole1 --user=tom
clusterrolebinding.rbac.authorization.k8s.io/clusterrolebinding1 created

root@k8scludes1:~/minsvcbug# kubectl get clusterrolebinding | grep clusterrolebinding1
clusterrolebinding1                                    ClusterRole/clusterole1                                                            21s

创建两条psp规则，一条PSP规则禁止创建特权用户pod，另外一条PSP规则允许创建特权用户pod。

编辑psp-allowpri-pod规则，psp-allowpri-pod规则允许创建特权用户pod，privileged: true。

root@k8scludes1:~/minsvcbug# vim psp-allowpri-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-allowpri-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-allowpri-pod
spec: 
  #allow privileged pods!
  privileged: true
  #hostNetwork: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
    #rule: MustRunAs
    #ranges:
    #- min: 1000
    #  max: 2000
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - '*'
  #- 'emptyDir'
  #- 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"
  #hostPorts:
  #- min: 1000
  #  max: 8090

编辑psp-denypri-pod规则，psp-allowpri-pod规则禁止创建特权用户pod，privileged: false。

root@k8scludes1:~/minsvcbug# vim psp-denypri-pod.yaml 

root@k8scludes1:~/minsvcbug# cat psp-denypri-pod.yaml 
apiVersion: policy/v1beta1 
kind: PodSecurityPolicy 
metadata: 
  name: psp-denypri-pod
spec: 
  #allow privileged pods!
  privileged: false
  #hostNetwork: true
  # The rest fills in some required fields. 
  seLinux: 
    rule: RunAsAny 
  supplementalGroups: 
    rule: RunAsAny 
  runAsUser: 
    rule: RunAsAny 
    #rule: MustRunAs
    #ranges:
    #- min: 1000
    #  max: 2000
  fsGroup: 
    rule: RunAsAny 
  volumes:
  - '*'
  #- 'emptyDir'
  #- 'projected'
    #allowedHostPaths:
    #- pathPrefix: "/tmp"
  #hostPorts:
  #- min: 1000
  #  max: 8090

创建PSP规则。

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-allowpri-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-allowpri-pod created

root@k8scludes1:~/minsvcbug# kubectl apply -f psp-denypri-pod.yaml 
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp-denypri-pod created

root@k8scludes1:~/minsvcbug# kubectl get psp -o wide
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME               PRIV    CAPS      SELINUX    RUNASUSER   FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
controller         false             RunAsAny   MustRunAs   MustRunAs   MustRunAs   true             configMap,secret,emptyDir
psp-allowpri-pod   true              RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
psp-denypri-pod    false             RunAsAny   RunAsAny    RunAsAny    RunAsAny    false            *
speaker            true    NET_RAW   RunAsAny   RunAsAny    RunAsAny    RunAsAny    true             configMap,secret,emptyDir

为了达到tom用户在所有的namespace里都不能创建特权用户pod的效果：需要通过clusterrole和clusterrolebinding把psp-denypri-pod规则授权给tom用户。

现在的clusterrole，clusterrolebinding如下：

root@k8scludes1:~/minsvcbug# kubectl get clusterrole | grep cluste
cert-manager-controller-clusterissuers                                 2022-04-24T17:07:47Z
cluster-admin                                                          2022-04-16T14:57:55Z
clusterole1                                                            2022-05-25T07:31:28Z
clusterrole--psp-mount-rule                                            2022-05-21T09:42:10Z
system:controller:clusterrole-aggregation-controller                   2022-04-16T14:57:55Z

root@k8scludes1:~/minsvcbug# kubectl get clusterrolebinding | grep cluster
cert-manager-controller-clusterissuers                 ClusterRole/cert-manager-controller-clusterissuers                                 30d
cluster-admin                                          ClusterRole/cluster-admin                                                          38d
clusterrolebinding1                                    ClusterRole/clusterole1                                                            30m
system:controller:clusterrole-aggregation-controller   ClusterRole/system:controller:clusterrole-aggregation-controller                   38d

创建clusterrole，角色clusterrole--psp-denypri-rule对名为psp-denypri-pod的psp规则有use权限。

root@k8scludes1:~/minsvcbug# kubectl create clusterrole clusterrole--psp-denypri-rule --verb=use --resource=psp --resource-name=psp-denypri-pod
clusterrole.rbac.authorization.k8s.io/clusterrole--psp-denypri-rule created

查看clusterrole--psp-denypri-rule的描述信息。

root@k8scludes1:~/minsvcbug# kubectl describe clusterrole clusterrole--psp-denypri-rule
Name:         clusterrole--psp-denypri-rule
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources                   Non-Resource URLs  Resource Names     Verbs
  ---------                   -----------------  --------------     -----
  podsecuritypolicies.policy  []                 [psp-denypri-pod]  [use]

把角色clusterrole--psp-denypri-rule绑定给tom用户，clusterrole--psp-denypri-rule和clusterrolebinding-clusterrole-denypri-psp的目的是让tom具有psp-denypri-pod规则的use权限，这样tom在所有的命名空间就不能创建特权用户pod了。

root@k8scludes1:~/minsvcbug# kubectl create clusterrolebinding clusterrolebinding-clusterrole-denypri-psp --clusterrole=clusterrole--psp-denypri-rule --user=tom
clusterrolebinding.rbac.authorization.k8s.io/clusterrolebinding-clusterrole-denypri-psp created

在客户端进行验证，首先测试在多个namespace创建和删除pod。

编辑pod配置文件，这是非特权用户pod。

[root@etcd2 minsvcbug]# vim podnotpri.yaml 

[root@etcd2 minsvcbug]# cat podnotpri.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
    #ports:
    #- name: http
    #  containerPort: 80
    #  hostPort: 8080
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

tom用户在ns1，ns2，ns3都能创建非特权pod。

[root@etcd2 minsvcbug]# kubectl apply -f podnotpri.yaml -n ns1
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -n ns1
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          12s

[root@etcd2 minsvcbug]# kubectl apply -f podnotpri.yaml -n ns2
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -n ns2
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          7s

[root@etcd2 minsvcbug]# kubectl apply -f podnotpri.yaml -n ns3
pod/podtest created

tom用户在ns1，ns2，ns3都能删除pod。

[root@etcd2 minsvcbug]# kubectl delete -f podnotpri.yaml -n ns1
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl delete -f podnotpri.yaml -n ns2
pod "podtest" deleted

#这样tom用户就可以在ns1，ns2，ns3创建和删除pod了
[root@etcd2 minsvcbug]# kubectl delete -f podnotpri.yaml -n ns3
pod "podtest" deleted

编辑pod配置文件，securityContext:privileged: true 表示创建特权用户pod。

[root@etcd2 minsvcbug]# vim podpri.yaml 

[root@etcd2 minsvcbug]# cat podpri.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    #securityContext:privileged: true 表示创建特权用户pod
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
    #ports:
    #- name: http
    #  containerPort: 80
    #  hostPort: 8080
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

可以发现，tom用户可以在ns1，ns2，ns3命名空间创建pod，但是不能创建特权pod。

[root@etcd2 minsvcbug]# kubectl apply -f podpri.yaml -n ns1
Error from server (Forbidden): error when creating "podpri.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

[root@etcd2 minsvcbug]# kubectl apply -f podpri.yaml -n ns2
Error from server (Forbidden): error when creating "podpri.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

[root@etcd2 minsvcbug]# kubectl apply -f podpri.yaml -n ns3
Error from server (Forbidden): error when creating "podpri.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

创建clusterrole，角色clusterrole--psp-allowpri-rule对名为psp-allowpri-pod的psp规则有use权限。

root@k8scludes1:~/minsvcbug# kubectl create clusterrole clusterrole--psp-allowpri-rule --verb=use --resource=psp --resource-name=psp-allowpri-pod
clusterrole.rbac.authorization.k8s.io/clusterrole--psp-allowpri-rule created

把角色clusterrole--psp-allowpri-rule绑定给tom用户，这样tom用户既可以访问psp-allowpri-pod规则，也可以访问psp-denypri-pod规则。

root@k8scludes1:~/minsvcbug# kubectl create rolebinding rolebinding-clusterrole-allowpri-psp --clusterrole=clusterrole--psp-allowpri-rule --user=tom
rolebinding.rbac.authorization.k8s.io/rolebinding-clusterrole-allowpri-psp created

删除rolebinding-clusterrole-allowpri-psp，因为没有指定是哪个命名空间，不指定的话默认是当前命名空间。

root@k8scludes1:~/minsvcbug# kubectl delete rolebinding rolebinding-clusterrole-allowpri-psp
rolebinding.rbac.authorization.k8s.io "rolebinding-clusterrole-allowpri-psp" deleted

重新进行角色绑定，这样tom用户在ns1命名空间既可以访问psp-allowpri-pod规则，也可以访问psp-denypri-pod规则。

注意：对于PSP规则，拒绝和允许同时出现的话，允许优先级比较高。

root@k8scludes1:~/minsvcbug# kubectl create rolebinding rolebinding-clusterrole-allowpri-psp --clusterrole=clusterrole--psp-allowpri-rule --user=tom -n ns1 
rolebinding.rbac.authorization.k8s.io/rolebinding-clusterrole-allowpri-psp created

下面在客户端进行测试。

编辑pod配置文件，下面的配置文件是为了创建特权用户pod。

[root@etcd2 minsvcbug]# cat podpri.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  terminationGracePeriodSeconds: 0
  volumes:
  - name: v1
    emptyDir: {}
  containers:
  - image: hub.c.163.com/library/nginx:latest
    imagePullPolicy: IfNotPresent
    name: podtest
    #创建特权pod
    securityContext:
      privileged: true
    resources: {}
    volumeMounts:
    - name: v1
      mountPath: /data
    #ports:
    #- name: http
    #  containerPort: 80
    #  hostPort: 8080
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

在ns1命名空间创建特权pod成功。

[root@etcd2 minsvcbug]# kubectl apply -f podpri.yaml -n ns1
pod/podtest created

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n ns1
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          12s   10.244.218.148   k8scludes2   <none>           <none>

在ns2，ns3命名空间创建特权pod失败。

[root@etcd2 minsvcbug]# kubectl apply -f podpri.yaml -n ns2
Error from server (Forbidden): error when creating "podpri.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

[root@etcd2 minsvcbug]# kubectl apply -f podpri.yaml -n ns3
Error from server (Forbidden): error when creating "podpri.yaml": pods "podtest" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

这样就实现了tom用户可以在ns1命名空间创建特权用户pod，在其他命名空间都不可以创建特权pod的需求。

清理一下pod。

[root@etcd2 minsvcbug]# kubectl delete -f podpri.yaml -n ns1
pod "podtest" deleted

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n ns1
No resources found in ns1 namespace.

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n ns2
No resources found in ns2 namespace.

[root@etcd2 minsvcbug]# kubectl get pod -o wide -n ns3
No resources found in ns3 namespace.

十三.PSP规则实战2:创建deployment

现在没有deploy，使用hub.c.163.com/library/nginx:latest镜像创建一个deploy，先生成yaml文件。关于deploy的详细操作，请查看博客《Kubernetes(k8s)控制器(一)：deployment》。

root@k8scludes1:~/minsvcbug# kubectl get deploy
No resources found in minsvcbug namespace.

root@k8scludes1:~/minsvcbug# kubectl create deployment nginxdeploy --image=hub.c.163.com/library/nginx:latest --dry-run=client -o yaml >nginxdeploy.yaml

生成的deploy配置文件如下：

root@k8scludes1:~/minsvcbug# vim nginxdeploy.yaml 

root@k8scludes1:~/minsvcbug# cat nginxdeploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  #这个labels是deployment的标签
  labels:
    app: nginxdeploy
  #deployment名字
  name: nginxdeploy
spec:
  #replicas: 1 表示副本数为1，只生成一个pod
  replicas: 1
  #selector标签选择器
  selector:
    matchLabels:
      app: nginxdeploy
  strategy: {}
  #template下面是pod的模板
  template:
    metadata:
      creationTimestamp: null
      #这个labels表示pod标签
      labels:
        app: nginxdeploy
    spec:
      #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
      terminationGracePeriodSeconds: 0
      containers:
      - image: hub.c.163.com/library/nginx:latest
        #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
        imagePullPolicy: IfNotPresent
        name: nginx
        resources: {}
status: {}

创建deploy。

root@k8scludes1:~/minsvcbug# kubectl apply -f nginxdeploy.yaml 
deployment.apps/nginxdeploy created

发现deploy不正常。

root@k8scludes1:~/minsvcbug# kubectl get deploy -o wide
NAME          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                               SELECTOR
nginxdeploy   0/1     0            0           18s   nginx        hub.c.163.com/library/nginx:latest   app=nginxdeploy

pod也没有生成。

root@k8scludes1:~/minsvcbug# kubectl get pod -o wide
No resources found in minsvcbug namespace.

注意：当我们创建一个deploy，比如nginxdeploy，则系统会为我们创建一个replicaset，名字为nginxdeploy-XXX，我们在deploy里指定的副本，本质上是由replicaset创建出来的，pod名字为nginxdeploy-XXX-yyyy。

可以发现，deployment和replicaset都没有ready。

root@k8scludes1:~/minsvcbug# kubectl get all -o wide
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                               SELECTOR
deployment.apps/nginxdeploy   0/1     0            0           7m17s   nginx        hub.c.163.com/library/nginx:latest   app=nginxdeploy

NAME                                     DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                               SELECTOR
replicaset.apps/nginxdeploy-5d458fcbfc   1         0         0       7m17s   nginx        hub.c.163.com/library/nginx:latest   app=nginxdeploy,pod-template-hash=5d458fcbfc

在kube-system命名空间里有很多个给控制器使用的sa。

root@k8scludes1:~/minsvcbug# kubectl get sa -n kube-system
NAME                                 SECRETS   AGE
attachdetach-controller              1         38d
bootstrap-signer                     1         38d
calico-kube-controllers              1         38d
calico-node                          1         38d
certificate-controller               1         38d
......
persistent-volume-binder             1         38d
pod-garbage-collector                1         38d
pv-protection-controller             1         38d
pvc-protection-controller            1         38d
replicaset-controller                1         38d
replication-controller               1         38d
resourcequota-controller             1         38d
root-ca-cert-publisher               1         38d
service-account-controller           1         38d
service-controller                   1         38d
statefulset-controller               1         38d
token-cleaner                        1         38d
ttl-after-finished-controller        1         38d
ttl-controller                       1         38d

只查看和controller有关的SA。

root@k8scludes1:~/minsvcbug# kubectl get sa -n kube-system | egrep '*controller'
attachdetach-controller              1         38d
calico-kube-controllers              1         38d
certificate-controller               1         38d
clusterrole-aggregation-controller   1         38d
cronjob-controller                   1         38d
daemon-set-controller                1         38d
deployment-controller                1         38d
disruption-controller                1         38d
endpoint-controller                  1         38d
endpointslice-controller             1         38d
endpointslicemirroring-controller    1         38d
ephemeral-volume-controller          1         38d
expand-controller                    1         38d
job-controller                       1         38d
namespace-controller                 1         38d
node-controller                      1         38d
pv-protection-controller             1         38d
pvc-protection-controller            1         38d
replicaset-controller                1         38d
replication-controller               1         38d
resourcequota-controller             1         38d
service-account-controller           1         38d
service-controller                   1         38d
statefulset-controller               1         38d
ttl-after-finished-controller        1         38d
ttl-controller                       1         38d

控制器创建pod的过程：在kube-system里有很多个给控制器使用的sa,启用PSP之后，现在控制器的SA都访问不了PSP规则，就创建不了pod,只有控制器的sa能访问PSP规则，才能正常创建pod。

现在需要给控制器SA进行授权，使控制器能访问PSP规则。

进行角色绑定，表示kube-system命名空间里的所有serviceaccounts都可以访问psp-allowpri-pod规则。

root@k8scludes1:~/minsvcbug# kubectl get clusterrole | grep cluster
cert-manager-controller-clusterissuers                                 2022-04-24T17:07:47Z
cluster-admin                                                          2022-04-16T14:57:55Z
clusterole1                                                            2022-05-25T07:31:28Z
clusterrole--psp-allowpri-rule                                         2022-05-25T08:57:00Z
clusterrole--psp-denypri-rule                                          2022-05-25T08:09:01Z
system:controller:clusterrole-aggregation-controller                   2022-04-16T14:57:55Z

root@k8scludes1:~/minsvcbug# kubectl create rolebinding rolebind-allowpri-psp-kube-system --clusterrole=clusterrole--psp-allowpri-rule --group=system:serviceaccounts -n kube-system --dry-run=client -o yaml > rolebind-allowpri-kube-system.yaml

root@k8scludes1:~/minsvcbug# kubectl apply -f rolebind-allowpri-kube-system.yaml 
rolebinding.rbac.authorization.k8s.io/rolebind-allowpri-psp-kube-system created

删除创建失败的nginxdeploy。

root@k8scludes1:~/minsvcbug# kubectl get deployments
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
nginxdeploy   0/1     0            0           3h13m

root@k8scludes1:~/minsvcbug# kubectl delete deploy nginxdeploy 
deployment.apps "nginxdeploy" deleted

root@k8scludes1:~/minsvcbug# kubectl get deployments
No resources found in minsvcbug namespace.

再次创建deploy。

root@k8scludes1:~/minsvcbug# kubectl apply -f nginxdeploy.yaml 
deployment.apps/nginxdeploy created

deploy还是没有创建成功。

root@k8scludes1:~/minsvcbug# kubectl get deployments
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
nginxdeploy   0/1     0            0           8s

root@k8scludes1:~/minsvcbug# kubectl get pod
No resources found in minsvcbug namespace.

查看nginxdeploy描述信息：replicaset-controller Error creating: pods "nginxdeploy-5d458fcbfc-" is forbidden: PodSecurityPolicy: unable to admit pod: []，可以发现权限还是不够。

root@k8scludes1:~/minsvcbug# kubectl describe deployment nginxdeploy 
Name:                   nginxdeploy
Namespace:              minsvcbug
......
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  34s (x14 over 75s)  replicaset-controller  Error creating: pods "nginxdeploy-5d458fcbfc-" is forbidden: PodSecurityPolicy: unable to admit pod: []

kubectl get ev查看事件。

root@k8scludes1:~/minsvcbug# kubectl get ev
LAST SEEN   TYPE      REASON              OBJECT                              MESSAGE
11m         Warning   FailedCreate        replicaset/nginxdeploy-5d458fcbfc   Error creating: pods "nginxdeploy-5d458fcbfc-" is forbidden: PodSecurityPolicy: unable to admit pod: []
101s        Warning   FailedCreate        replicaset/nginxdeploy-5d458fcbfc   Error creating: pods "nginxdeploy-5d458fcbfc-" is forbidden: PodSecurityPolicy: unable to admit pod: []
4m25s       Normal    ScalingReplicaSet   deployment/nginxdeploy              Scaled up replica set nginxdeploy-5d458fcbfc to 1

删除nginxdeploy。

root@k8scludes1:~/minsvcbug# kubectl get deployments
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
nginxdeploy   0/1     0            0           5m18s

root@k8scludes1:~/minsvcbug# kubectl delete deploy nginxdeploy 
deployment.apps "nginxdeploy" deleted

root@k8scludes1:~/minsvcbug# kubectl get deployments
No resources found in minsvcbug namespace.

root@k8scludes1:~/minsvcbug# kubectl get all
No resources found in minsvcbug namespace.

编辑ClusterRoleBinding配置文件，把RoleBinding变为ClusterRoleBinding，使其全局生效，刚才的角色绑定rolebind-allowpri-psp-kube-system是错误的。

root@k8scludes1:~/minsvcbug# vim rolebind-allowpri-kube-system.yaml 

root@k8scludes1:~/minsvcbug# cat rolebind-allowpri-kube-system.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: clusterrolebind-allowpri-psp-kube-system
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: clusterrole--psp-allowpri-rule
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts

删除rolebind-allowpri-psp-kube-system。

root@k8scludes1:~/minsvcbug# kubectl get rolebinding -n kube-system | grep rolebind-allowpri-psp-kube-system
rolebind-allowpri-psp-kube-system                   ClusterRole/clusterrole--psp-allowpri-rule            12m

root@k8scludes1:~/minsvcbug# kubectl delete rolebinding rolebind-allowpri-psp-kube-system -n kube-system
rolebinding.rbac.authorization.k8s.io "rolebind-allowpri-psp-kube-system" deleted

root@k8scludes1:~/minsvcbug# kubectl get rolebinding -n kube-system | grep rolebind-allowpri-psp-kube-system

创建clusterrolebinding。

root@k8scludes1:~/minsvcbug# kubectl apply -f rolebind-allowpri-kube-system.yaml 
clusterrolebinding.rbac.authorization.k8s.io/clusterrolebind-allowpri-psp-kube-system created

root@k8scludes1:~/minsvcbug# kubectl get clusterrolebinding -n kube-system | egrep clusterrolebind-allowpri-psp-kube-system
clusterrolebind-allowpri-psp-kube-system               ClusterRole/clusterrole--psp-allowpri-rule                                         45s

创建deployment。

root@k8scludes1:~/minsvcbug# kubectl apply -f nginxdeploy.yaml 
deployment.apps/nginxdeploy created

现在deploy就创建成功了。

root@k8scludes1:~/minsvcbug# kubectl get all -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
pod/nginxdeploy-5d458fcbfc-rs42b   1/1     Running   0          12s   10.244.218.160   k8scludes2   <none>           <none>

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                               SELECTOR
deployment.apps/nginxdeploy   1/1     1            1           12s   nginx        hub.c.163.com/library/nginx:latest   app=nginxdeploy

NAME                                     DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                               SELECTOR
replicaset.apps/nginxdeploy-5d458fcbfc   1         1         1       12s   nginx        hub.c.163.com/library/nginx:latest   app=nginxdeploy,pod-template-hash=5d458fcbfc

root@k8scludes1:~/minsvcbug# kubectl get pod
NAME                           READY   STATUS    RESTARTS   AGE
nginxdeploy-5d458fcbfc-rs42b   1/1     Running   0          71s

clusterrolebinding全局生效，所有命名空间都能看到。

root@k8scludes1:~/minsvcbug# kubectl get clusterrolebinding | egrep clusterrolebind-allowpri-psp-kube-system
clusterrolebind-allowpri-psp-kube-system               ClusterRole/clusterrole--psp-allowpri-rule                                         4m43s

root@k8scludes1:~/minsvcbug# kubectl get clusterrolebinding -n default | egrep clusterrolebind-allowpri-psp-kube-system
clusterrolebind-allowpri-psp-kube-system               ClusterRole/clusterrole--psp-allowpri-rule                                         4m53s

删除nginxdeploy。

root@k8scludes1:~/minsvcbug# kubectl delete deploy nginxdeploy 
deployment.apps "nginxdeploy" deleted

root@k8scludes1:~/minsvcbug# kubectl get all
No resources found in minsvcbug namespace.

十四.总结

PodSecurityPolicy 是 Kubernetes 集群安全的重要组成部分。通过定义 Pod 的安全策略，管理员可以确保集群中的 Pod 遵循特定的安全标准，从而降低潜在的安全风险。本文介绍了 PSP 的概念、重要性以及如何在 Kubernetes 集群中实施。希望这些信息能够帮助您更好地保护您的 Kubernetes 集群。