Kubernetes 部署 nfs-subdir-external-provisioner

概述

官方GitHub及参考文档:GitHub - kubernetes-sigs/nfs-subdir-external-provisioner: Dynamic sub-dir volume provisioner on a remote NFS server.

部署nfs-subdir-external-provisioner提供StorageClass服务

步骤

nfs 服务器准备

/etc/exports

# cat /etc/exports
/nfsshare *(rw,no_root_squash,sync)

install nfs-utils

yum -y install nfs-utils
systemctl enable --now nfs

namespace

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: nfs-server
  name: nfs-server
spec:
  finalizers:
    - kubernetes

class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true"
  # 定义路径格式
  pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"

不同的呈现方式

rbac

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs-server
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs-server
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          # image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          image: registry.cn-beijing.aliyuncs.com/xngczl/nfs-subdir-external-provisione:v4.0.0
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
              # 设置高可用允许选举
            - name: ENABLE_LEADER_ELECTION
              value: "True"
            - name: NFS_SERVER
              value: 192.168.164.16
            - name: NFS_PATH
              value: /nfsshare
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.164.16
            path: /nfsshare

验证命令

kubectl logs -f --tail=20 nfs-client-provisioner-5856c5fc68-2r4k2  -n nfs-server                                                                                                                                      

kubectl get sc

 测试

pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs-client"
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

pod 

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: busybox:stable
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim

Error

No such file or directory

  Warning  FailedMount  15s (x8 over 79s)  kubelet            MountVolume.SetUp failed for volume "nfs-client-r                                                                                                            oot" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 192.168.164.16:/nfsshare /etc/cni/net.d/pods/67154eef-96d4-46cc-af01-653496a4f1a7/vo                                                                                                            lumes/kubernetes.io~nfs/nfs-client-root
Output: mount.nfs: mounting 192.168.164.16:/nfsshare failed, reason given by server: No such file or directory

报错原因: 配置文件编写错误：

修改正确的配置文件/etc/exports后，需要重启nfs服务生效。

mount挂载文件，有坏超级块

Mounting arguments: --description=Kubernetes transient mount for /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs 172.16.41.7:/data/nfs_storage /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root

Output: Running scope as unit: run-rdcc7cfa6560845969628fc551606e69d.scope

mount: /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program.

  Warning  FailedMount  10s  kubelet, node1.ayunw.cn  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32

Mounting command: systemd-run

报错原因: 各个Kubernetes节点需要安装nfs-utils

yum -y install nfs-utils 执行安装即可

waiting for a volume to be created

  Normal  ExternalProvisioning  8s (x17 over 3m42s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "k8s-sigs.io/nfs-subdir-external-provisioner" or manually created by system administrator

pvc 一直处于pending状态，事件一直处于等待。

检查class.yaml文件和deployment文件

参考文档

mount挂载文件，有坏超级块（解决方案） - 代码先锋网 (codeleading.com)

k8s 使用新版NFS Provisioner配置subdir - 知乎 (zhihu.com)

k8s-1.22.3版本中使用持久化卷之StorageClass+NFS (zhihu.com)

k8s-1.22.3版本部署持久化存储之StorageClass+NFS_nfs部署storageclass_归海听雪的博客-CSDN博客

K8s生产环境常见问题处理、答疑(连载、不定期更新)-阿里云开发者社区 (aliyun.com)

Releases · kubernetes-sigs/nfs-subdir-external-provisioner (github.com)

k8s学习： 部署动态 pvc（nfs-subdir-external-provisioner）_动态pvc_哈哈虎123的博客-CSDN博客