目录
3.2、使用 Deployment 来创建 NFS Provisioner
3.7、进入 Pod 在挂载目录 /mnt 下写一个文件,然后查看 NFS 服务器上是否存在该文件
pv是集群中的资源,PVC是对这些资源的请求,也是对资源的索引检查
PV和PVC之间的相互作用遵循这个生命周期:
Provisioning(配置)----Binding(绑定)----Using(使用)----Releasing(释放)----Recycling(回收)
有三种回收策略,Retain、Delete和Recycle。
查看PV的定义方式
kubectl explain pv
- FIELDS:
- apiVersion: v1
- kind: PersistentVolume
- metadata: #由于 PV 是集群级别的资源,即 PV 可以跨 namespace 使用,所以 PV 的 metadata 中不用配置 namespace
- name:
- spec
查看PV定义的规格
kubectl explain pv.spec
- spec:
- nfs:(定义存储类型)
- path:(定义挂载卷路径)
- server:(定义服务器名称)
- accessModes:(定义访问模型,有以下三种访问模型,以列表的方式存在,也就是说可以定义多个访问模式)
- - ReadWriteOnce #(RWO)存储可读可写,但只支持被单个 Pod 挂载
- - ReadOnlyMany #(ROX)存储可以以只读的方式被多个 Pod 挂载
- - ReadWriteMany #(RWX)存储可以以读写的方式被多个 Pod 共享
- #nfs 支持全部三种;iSCSI 不支持 ReadWriteMany(iSCSI 就是在 IP 网络上运行 SCSI 协议的一种网络存储技术);HostPath 不支持 ReadOnlyMany 和 ReadWriteMany。
- capacity:(定义存储能力,一般用于设置存储空间)
- storage: 2Gi (指定大小)
- storageClassName: (自定义存储类名称,此配置用于绑定具有相同类别的PVC和PV)
- persistentVolumeReclaimPolicy: Retain #回收策略(Retain/Delete/Recycle)
- #Retain(保留):当删除与之绑定的PVC时候,这个PV被标记为released(PVC与PV解绑但还没有执行回收策略)且之前的数据依然保存在该PV上,但是该PV不可用,需要手动来处理这些数据并删除该PV。
- #Delete(删除):删除与PV相连的后端存储资源(只有 AWS EBS, GCE PD, Azure Disk 和 Cinder 支持)
- #Recycle(回收):删除数据,效果相当于执行了 rm -rf /thevolume/* (只有 NFS 和 HostPath 支持)
查看PVC的定义方式
kubectl explain pvc
- KIND: PersistentVolumeClaim
- VERSION: v1
- FIELDS:
- apiVersion <string>
- kind <string>
- metadata <Object>
- spec <Object>
PV和PVC的spec关键字段要匹配,比如存储(storage)大小、访问模式(accessModes)、存储类名称(storageClassName)
- kubectl explain pvc.spec
- spec:
- accessModes: (定义访问模式,必须是PV的访问模式的子集)
- resources:
- requests:
- storage: (定义申请资源的大小)
- storageClassName: (定义存储类名称,此配置用于绑定具有相同类别的PVC和PV)
这里又两种PV的提供方式:静态或者动态
静态---》直接固定存储空间
动态---》通过存储类进行动态创建存储空间
nfs-server | k8s-master(192.168.130.70) |
---|---|
nfs-client | k8s-node1(192.168.130.60),k8s-node2(192.168.130.50) |
yum install -y nfs-utils rpcbind
mkdir -p /data/volumes/v{1..5}
- vim /etc/exports
- /data/volumes/v1 192.168.130.0/24(rw,root_squash,sync,all_squash)
- /data/volumes/v2 192.168.130.0/24(rw,root_squash,sync,all_squash)
- /data/volumes/v3 192.168.130.0/24(rw,root_squash,sync,all_squash)
- /data/volumes/v4 192.168.130.0/24(rw,root_squash,sync,all_squash)
- /data/volumes/v5 192.168.130.0/24(rw,root_squash,sync,all_squash)
-
- exportfs -arv
-
- showmount -e
- #手动加载 NFS 共享服务时,应该先启动 rpcbind,再启动 nfs
- systemctl start rpcbind && systemctl enable rpcbind
- systemctl start nfs && systemctl enable nfs
-
- #查看 rpcbind 端口是否开启,rpcbind 服务默认使用 tcp 端口 111
- netstat -anpt | grep rpcbind
- vim pv-demo.yaml
-
-
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv001
- labels:
- name: pv001
- spec:
- nfs:
- path: /data/volumes/v1
- server: stor01 #改为本机IP或hosts指定域名
- accessModes: ["ReadWriteMany","ReadWriteOnce"]
- capacity:
- storage: 1Gi
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv002
- labels:
- name: pv002
- spec:
- nfs:
- path: /data/volumes/v2
- server: stor01 #改为本机IP或hosts指定域名
- accessModes: ["ReadWriteOnce"]
- capacity:
- storage: 2Gi
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv003
- labels:
- name: pv003
- spec:
- nfs:
- path: /data/volumes/v3
- server: stor01 #改为本机IP或hosts指定域名
- accessModes: ["ReadWriteMany","ReadWriteOnce"]
- capacity:
- storage: 2Gi
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv004
- labels:
- name: pv004
- spec:
- nfs:
- path: /data/volumes/v4
- server: stor01 #改为本机IP或hosts指定域名
- accessModes: ["ReadWriteMany","ReadWriteOnce"]
- capacity:
- storage: 4Gi
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: pv005
- labels:
- name: pv005
- spec:
- nfs:
- path: /data/volumes/v5
- server: stor01 #改为本机IP或hosts指定域名
- accessModes: ["ReadWriteMany","ReadWriteOnce"]
- capacity:
- storage: 5Gi
kubectl apply -f pv-demo.yaml
- kubectl get pv
-
-
- NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
- pv001 1Gi RWO,RWX Retain Available 7s
- pv002 2Gi RWO Retain Available 7s
- pv003 2Gi RWO,RWX Retain Available 7s
- pv004 4Gi RWO,RWX Retain Available 7s
- pv005 5Gi RWO,RWX Retain Available 7s
- vim pod-vol-pvc.yaml
-
- apiVersion: v1
- kind: PersistentVolumeClaim
- metadata:
- name: mypvc
- namespace: default
- spec:
- accessModes: ["ReadWriteMany"]
- resources:
- requests:
- storage: 2Gi
- ---
- apiVersion: v1
- kind: Pod
- metadata:
- name: pod-vol-pvc
- namespace: default
- spec:
- containers:
- - name: myapp
- image: ikubernetes/myapp:v1
- volumeMounts:
- - name: html
- mountPath: /usr/share/nginx/html
- volumes:
- - name: html
- persistentVolumeClaim:
- claimName: mypvc
kubectl apply -f pod-vol-pvc.yaml
- kubectl get pv
-
- NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
- pv001 1Gi RWO,RWX Retain Available 19m
- pv002 2Gi RWO Retain Available 19m
- pv003 2Gi RWO,RWX Retain Bound default/mypvc 19m
- pv004 4Gi RWO,RWX Retain Available 19m
- pv005 5Gi RWO,RWX Retain Available 19m
-
- kubectl get pvc
-
- NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
- mypvc Bound pv003 2Gi RWO,RWX 22s
//在存储服务器上创建index.html,并写入数据,通过访问Pod进行查看,可以获取到相应的页面。
- cd /data/volumes/v3/
- echo "welcome to use pv3" > index.html
-
- kubectl get pods -o wide
-
- pod-vol-pvc 1/1 Running 0 3m 10.244.2.5 k8s-node02
-
- curl 10.244.2.5
-
- welcome to use pv3
Kubernetes 本身支持的动态 PV 创建不包括 NFS,所以需要使用外部存储卷插件分配PV。详见:存储类 | Kubernetes
卷插件称为 Provisioner(存储分配器),NFS 使用的是 nfs-client,这个外部卷插件会使用已经配置好的 NFS 服务器自动创建 PV。 Provisioner:用于指定 Volume 插件的类型,包括内置插件(如 kubernetes.io/aws-ebs)和外部插件(如 external-storage 提供的 ceph.com/cephfs)。
- vim nfs-client-rbac.yaml
- #创建 Service Account 账户,用来管理 NFS Provisioner 在 k8s 集群中运行的权限
-
-
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: nfs-client-provisioner
- ---
- #创建集群角色
- apiVersion: rbac.authorization.k8s.io/v1
- kind: ClusterRole
- metadata:
- name: nfs-client-provisioner-clusterrole
- rules:
- - apiGroups: [""]
- resources: ["persistentvolumes"]
- verbs: ["get", "list", "watch", "create", "delete"]
- - apiGroups: [""]
- resources: ["persistentvolumeclaims"]
- verbs: ["get", "list", "watch", "update"]
- - apiGroups: ["storage.k8s.io"]
- resources: ["storageclasses"]
- verbs: ["get", "list", "watch"]
- - apiGroups: [""]
- resources: ["events"]
- verbs: ["list", "watch", "create", "update", "patch"]
- - apiGroups: [""]
- resources: ["endpoints"]
- verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- ---
- #集群角色绑定
- apiVersion: rbac.authorization.k8s.io/v1
- kind: ClusterRoleBinding
- metadata:
- name: nfs-client-provisioner-clusterrolebinding
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- namespace: default
- roleRef:
- kind: ClusterRole
- name: nfs-client-provisioner-clusterrole
- apiGroup: rbac.authorization.k8s.io
kubectl apply -f nfs-client-rbac.yaml
NFS Provisione(即 nfs-client),有两个功能:一个是在 NFS 共享目录下创建挂载点(volume),另一个则是将 PV 与 NFS 的挂载点建立关联。
#由于 1.20 版本启用了 selfLink,所以 k8s 1.20+ 版本通过 nfs provisioner 动态生成pv会报错,解决方法如下:
vim /etc/kubernetes/manifests/kube-apiserver.yaml
- #kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
- #kubectl delete pods kube-apiserver -n kube-system
- 修改完等待重启就行,上面命令不需要执行
- kubectl get pods -n kube-system | grep apiserver
- vim nfs-client-provisioner.yaml
- kind: Deployment
- apiVersion: apps/v1
- metadata:
- name: nfs-client-provisioner
- spec:
- replicas: 1
- selector:
- matchLabels:
- app: nfs-client-provisioner
- strategy:
- type: Recreate
- template:
- metadata:
- labels:
- app: nfs-client-provisioner
- spec:
- serviceAccountName: nfs-client-provisioner #指定Service Account账户
- containers:
- - name: nfs-client-provisioner
- image: quay.io/external_storage/nfs-client-provisioner:latest
- imagePullPolicy: IfNotPresent
- volumeMounts:
- - name: nfs-client-root
- mountPath: /persistentvolumes
- env:
- - name: PROVISIONER_NAME
- value: nfs-storage #配置provisioner的Name,确保该名称与StorageClass资源中的provisioner名称保持一致
- - name: NFS_SERVER
- value: 192.168.130.70 #配置绑定的nfs服务器
- - name: NFS_PATH
- value: /data #配置绑定的nfs服务器目录
- volumes: #申明nfs数据卷
- - name: nfs-client-root
- nfs:
- server: 192.168.130.70
- path: /data
kubectl apply -f nfs-client-provisioner.yaml
负责建立 PVC 并调用 NFS provisioner 进行预定的工作,并让 PV 与 PVC 建立关联
- vim nfs-client-storageclass.yaml
-
- apiVersion: storage.k8s.io/v1
- kind: StorageClass
- metadata:
- name: nfs-client-storageclass
- provisioner: nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致
- parameters:
- archiveOnDelete: "false" #false表示在删除PVC时不会对数据进行存档,即删除数据
kubectl apply -f nfs-client-storageclass.yaml
- vim test-pvc-pod.yaml
-
- apiVersion: v1
- kind: PersistentVolumeClaim
- metadata:
- name: test-nfs-pvc
- spec:
- accessModes:
- - ReadWriteMany
- storageClassName: nfs-client-storageclass #关联storageclass对象
- resources:
- requests:
- storage: 1Gi
- ---
- apiVersion: v1
- kind: Pod
- metadata:
- name: test-storageclass-pod
- spec:
- containers:
- - name: busybox
- image: busybox:latest
- imagePullPolicy: IfNotPresent
- command:
- - "/bin/sh"
- - "-c"
- args:
- - "sleep 3600"
- volumeMounts:
- - name: nfs-pvc
- mountPath: /mnt
- restartPolicy: Never
- volumes:
- - name: nfs-pvc
- persistentVolumeClaim:
- claimName: test-nfs-pvc #与PVC名称一致
kubectl apply -f test-pvc-pod.yaml
ls /data/volumes/v5
- kubectl exec -it test-storageclass-pod sh
-
- #进入挂载目录
- cd /mnt
- echo "this is test" > 1.txt
cat /data/volumes/v5/default-test-nfs-pvc-pvc-2436629e-ffca-4c1e-9172-123d5e9aa332/1.txt
发现DNS服务器上有这个文件,成功了