• K8S上安装LongHorn(分布式块存储) --use

    要在 Kubernetes上安装 LongHorn，您可以按照以下步骤进行操作：

    准备工作

    参考

    安装要求

    Each node in the Kubernetes cluster where Longhorn is installed must fulfill the following requirements:

    • A container runtime compatible with Kubernetes (Docker v1.13+, containerd v1.3.7+, etc.)
    • Kubernetes >= v1.21
    • open-iscsi is installed, and the iscsid daemon is running on all the nodes. This is necessary, since Longhorn relies on iscsiadm on the host to provide persistent volumes to Kubernetes. For help installing open-iscsi, refer to this section.
    • RWX support requires that each node has a NFSv4 client installed.
    • The host filesystem supports the file extents feature to store the data. Currently we support:
      • ext4
      • XFS
    • bash, curl, findmnt, grep, awk, blkid, lsblk must be installed.
    • Mount propagation must be enabled.

    The Longhorn workloads must be able to run as root in order for Longhorn to be deployed and operated properly.

    使用环境检查脚本

    1. # use
    2. curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/scripts/environment_check.sh | bash
    3.  
    4. # 参考
    5. curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.6.1/scripts/environment_check.sh | bash

    创建命名空间

    1. # vi longhorn-ns.yaml
    2. # kubectl apply -f longhorn-ns.yaml
    3. # kubectl delete -f longhorn-ns.yaml
    4. apiVersion: v1
    5. kind: Namespace
    6. metadata:
    7.   name: longhorn-system

    安装 open-iscsi(可选)

    1. # use
    2. apt-get install open-iscsi
    3.  
    4. systemctl enable iscsid --now
    5. systemctl status iscsid
    6.  
    7. modprobe iscsi_tcp
    8.  
    9. $. wget https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/prerequisite/longhorn-iscsi-installation.yaml
    10.  
    11. $. vi longhorn-iscsi-installation.yaml
    12.    # 添加如下内容
    13.     metadata:
    14.       namespace: longhorn-system
    15.    # nodeselector
    16.    spec:
    17.      template:
    18.        spec:
    19.          nodeSelector:
    20.            longhorn: deploy
    21. $. kubectl apply -f longhorn-iscsi-installation.yaml
    22. $. kubectl get DaemonSet longhorn-iscsi-installation -n longhorn-system
    23. $. kubectl get pod \
    24. -o wide \
    25. -n longhorn-system |\
    26.  grep longhorn-iscsi-installation
    27. $. kubectl delete -f longhorn-iscsi-installation.yaml
    28.  
    29. # 查看log
    30. kubectl logs longhorn-iscsi-installation-44h92 \
    31. -c iscsi-installation \
    32. -n longhorn-system
    33.  
    34. # ========================== 官网 =========================== #
    35. # SUSE and openSUSE: Run the following command:
    36. zypper install open-iscsi
    37.  
    38. # Debian and Ubuntu: Run the following command:
    39. apt-get install open-iscsi
    40.  
    41. # RHEL, CentOS, and EKS (EKS Kubernetes Worker AMI with AmazonLinux2 image): Run the following commands:
    42. yum --setopt=tsflags=noscripts install iscsi-initiator-utils
    43. echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi
    44. systemctl enable iscsid
    45. systemctl start iscsid
    46.  
    47. # 验证
    48. $. iscsiadm --version
    49.  
    50. # =============================== 参考 =============================== #
    51. # 前言：这里二选一要么直接安装命令，要么使用容器去进行安装，这里选择之间安装的命令。
    52. # open-iscsi
    53. ​kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.1/deploy/prerequisite/longhorn-iscsi-installation.yaml
    54. # nfs
    55. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.1/deploy/prerequisite/longhorn-nfs-installation.yaml

    安装 NFSv4 client(可选)

    1. # Check NFSv4.1 support is enabled in kernel
    2. cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_1
    3.  
    4. # Check NFSv4.2 support is enabled in kernel
    5. cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_2
    6.  
    7. ## The command used to install a NFSv4 client differs depending on the Linux distribution.
    8.  
    9.  # For Debian and Ubuntu, use this command:
    10.     apt-get install nfs-common
    11.  
    12.   # For RHEL, CentOS, and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:
    13.     yum install nfs-utils
    14.   
    15.   # For SUSE/OpenSUSE you can install a NFSv4 client via:
    16.     zypper install nfs-client
    17.  
    18. # For Talos Linux, the NFS client is part of the kubelet image maintained by the Talos team.
    19.  
    20. # We also provide an nfs installer to make it easier for users to install nfs-client automatically:
    21. # use
    22. $. wget https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/prerequisite/longhorn-nfs-installation.yaml
    23. $. vi longhorn-nfs-installation.yaml
    24.    # 添加如下内容
    25.     metadata:
    26.       namespace: longhorn-system
    27.    # nodeselector
    28.    spec:
    29.      template:
    30.        spec:
    31.          nodeSelector:
    32.            longhorn: deploy
    33. $. kubectl apply -f longhorn-nfs-installation.yaml
    34. $. kubectl get DaemonSet longhorn-nfs-installation -n longhorn-system
    35. $. kubectl get pod \
    36. -o wide \
    37. -n longhorn-system \
    38. | grep longhorn-nfs-installation
    39.  
    40. $. kubectl delete -f longhorn-nfs-installation.yaml
    41.  
    42. # ===================== 参考 ===================== #
    43. $. kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.6.1/deploy/prerequisite/longhorn-nfs-installation.yaml
    44.  
    45. # After the deployment, run the following command to check pods’ status of the installer:
    46. $. kubectl get pod | grep longhorn-nfs-installation
    47. NAME                                  READY   STATUS    RESTARTS   AGE
    48. longhorn-nfs-installation-t2v9v   1/1     Running   0          143m
    49. longhorn-nfs-installation-7nphm   1/1     Running   0          143m
    50. And also can check the log with the following command to see the installation result:
    51.  
    52. $. kubectl logs longhorn-nfs-installation-t2v9v -c nfs-installation
    53. ...
    54. nfs install successfully

    给节点打标签

    1. $. kubectl label node k8s-infra01 longhorn=deploy
    2. $. kubectl get no k8s-infra01 --show-labels
    3.  
    4. # 参考
    5. $. kubectl label node k8s-worker2 longhorn-
    6. $. kubectl label node k8s-infra01 longhorn=deploy
    7. $. kubectl label node k8s-infra01 longhorn-
    8. $. kubectl get no k8s-worker5 --show-labels
    9.  
    10. # ======================== 参考 ======================== #
    11. # 查询node标签
    12. # kubectl get no -o wide --show-labels | grep k8s-worker5
    13.  
    14. # 删除label
    15. # kubectl label nodes k8s-worker5 k8s-worker5-
    16.  
    17. # 修改label
    18. # kubectl label nodes k8s-worker5 longhorn=deploy --overwrite
    19. # or 
    20. # kubectl edit nodes k8s-worker5

    部署LongHorn

    官网地址: longhorn-官网安装

    下载longhorn.yaml

    1. # use
    2. $. wget https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
    3. $. cp longhorn.yaml longhorn.yaml-bak
    4.  
    5. # old
    6. $. kubectl apply \
    7. -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
    8.  
    9. # ========================= 参考 ======================== #
    10. # 官网地址: https://longhorn.io/docs/1.6.0/deploy/install/install-with-kubectl/

    vi longhorn.yaml

    注释掉其中的affinity

    查找 kind: Deployment, 并在注释掉其中的affinity：

    1. # 注释掉其中的affinity
    2. ...    
    3. spec:
    4.   template:
    5.     spec:
    6.       #affinity:
    7.       #  podAntiAffinity:
    8.       #    preferredDuringSchedulingIgnoredDuringExecution:
    9.       #    - weight: 1
    10.       #      podAffinityTerm:
    11.       #        labelSelector:
    12.       #          matchExpressions:
    13.       #          - key: app
    14.       #            operator: In
    15.       #            values:
    16.       #            - longhorn-ui
    17.       #        topologyKey: kubernetes.io/hostname
    18. ...

    设置Node Selector

    using kubectl to apply the deployment YAML, modify the node selector section for Longhorn Manager, Longhorn UI, and Longhorn Driver Deployer. Then apply the YAMl files.

    查找 kind: Deployment, 并在添加nodeselector：

    1. spec:
    2.   template:
    3.     spec:
    4.       # lpf
    5.       nodeSelector:
    6.         longhorn: deploy

    修改存储路径

    默认存储路径为：/var/lib/longhorn，可修改为自己想要路径
    找到名为 longhorn-manager的DaemonSet资源下的 spec.template.spec.[containers].[volumes].hostPath.path字段。

    修改pv为保留策略

    修改pv为保留策略，否则删除pvc的时候pv就会被删除。 
    1. data:
    2.   storageclass.yaml: |
    3.     kind: StorageClass
    4.     apiVersion: storage.k8s.io/v1
    5.     metadata:
    6.       name: longhorn
    7.       annotations:
    8.         storageclass.kubernetes.io/is-default-class: "true"
    9.     provisioner: driver.longhorn.io
    10.     allowVolumeExpansion: true
    11.     # 找到文件中的StorageClass.data.reclaimPolicy字段,修改为Retain
    12.     # reclaimPolicy: "Delete"
    13.     reclaimPolicy: "Retain"
    14.     volumeBindingMode: Immediate

    修改资源类型和端口

    修改longhorn-frontend 服务的svc资源类型和端口，便于web界面查看。

    1. kind: Service
    2. apiVersion: v1
    3. metadata:
    4.   labels:
    5.     app.kubernetes.io/name: longhorn
    6.     app.kubernetes.io/instance: longhorn
    7.     app.kubernetes.io/version: v1.3.2
    8.     app: longhorn-ui
    9.   name: longhorn-frontend
    10.   namespace: longhorn-system
    11. spec:
    12.   # 修改为 NodePort，默认为ClusterIP
    13.   #type: ClusterIP
    14.   type: NodePort
    15.   selector:
    16.     app: longhorn-ui
    17.   ports:
    18.   - name: http
    19.     port: 80
    20.     targetPort: http
    21.     # 修改端口号，默认为 null
    22.     #nodePort: null
    23.     nodePort: 32183

    开始部署

    1. $. kubectl apply -f longhorn.yaml
    2.  
    3. # 命令试执行(不会真的创建或修改任何集群中的资源。)
    4. $. kubectl --dry-run=client apply -f longhorn.yaml
    5.  
    6. # 查看部署效果
    7. $. kubectl get all -o wide -n longhorn-system
    8. $. kubectl get pod -o wide -n longhorn-system
    9.  
    10. # ds
    11. kubectl get ds -n longhorn-system
    12.  
    13. # ========================= 参考 ========================== #
    14. # 删除longhorn
    15. # 步骤1
    16. $. kubectl delete -f longhorn.yaml
    17. # 步骤2
    18. # 删除k8s各个node下的/var/lib/longhorn目录(若改为自定义目录, 则删除相关自定义目录即可)
    19.

    设置nodeselector

    给所有daemonset和deployment设置nodeselector.

    1. $. kubectl get deploy,ds,rs -n longhorn-system
    2. # 给下面所有资源设置 nodeselector
    3. # spec:
    4. #  template:
    5. #    spec:
    6. #      # lpf
    7. #      nodeSelector:
    8. #        longhorn: deploy
    9. # 设置命令:
    10. # kubectl edit deploy csi-attacher -n longhorn-system
    11. $. kubectl patch \
    12. deploy csi-attacher \
    13. -n longhorn-system \
    14. -p '{"spec": {"template": {"spec": {"nodeSelector": {"longhorn": "deploy"}}}}}'
    15.  
    16. # kubectl edit deploy csi-provisioner -n longhorn-system
    17. $. kubectl patch \
    18. deploy csi-provisioner \
    19. -n longhorn-system \
    20. -p '{"spec": {"template": {"spec": {"nodeSelector": {"longhorn": "deploy"}}}}}'
    21.  
    22. # kubectl edit deploy csi-resizer -n longhorn-system
    23. $. kubectl patch \
    24. deploy csi-resizer \
    25. -n longhorn-system \
    26. -p '{"spec": {"template": {"spec": {"nodeSelector": {"longhorn": "deploy"}}}}}'
    27.  
    28. # kubectl edit deploy csi-snapshotter -n longhorn-system
    29. $. kubectl patch \
    30. deploy csi-snapshotter \
    31. -n longhorn-system \
    32. -p '{"spec": {"template": {"spec": {"nodeSelector": {"longhorn": "deploy"}}}}}'
    33.  
    34. # kubectl edit ds engine-image-ei-acb7590c -n longhorn-system
    35. $. kubectl patch \
    36. ds engine-image-ei-b907910b \
    37. -n longhorn-system \
    38. -p '{"spec": {"template": {"spec": {"nodeSelector": {"longhorn": "deploy"}}}}}'
    39.  
    40. # kubectl edit ds longhorn-csi-plugin -n longhorn-system
    41. $. kubectl patch \
    42. ds longhorn-csi-plugin \
    43. -n longhorn-system \
    44. -p '{"spec": {"template": {"spec": {"nodeSelector": {"longhorn": "deploy"}}}}}'
    45.  
    46. # 验证
    47. # kubectl get all -o wide -n longhorn-system
    48.  
    49. NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
    50. deployment.apps/csi-attacher               3/3     3            3           49m
    51. deployment.apps/csi-provisioner            3/3     3            3           49m
    52. deployment.apps/csi-resizer                3/3     3            3           49m
    53. deployment.apps/csi-snapshotter            3/3     3            3           49m
    54. deployment.apps/longhorn-driver-deployer   1/1     1            1           49m
    55. deployment.apps/longhorn-ui                2/2     2            2           49m
    56.  
    57. NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR     AGE
    58. daemonset.apps/engine-image-ei-acb7590c      1         1         1       1            1           longhorn=deploy   49m
    59. daemonset.apps/longhorn-csi-plugin           9         9         9       9            9           <none>            49m
    60. daemonset.apps/longhorn-iscsi-installation   1         1         1       1            1           longhorn=deploy   49m
    61. daemonset.apps/longhorn-manager              1         1         1       1            1           longhorn=deploy   49m
    62. daemonset.apps/longhorn-nfs-installation     1         1         1       1            1           longhorn=deploy   48m
    63.  
    64. NAME                                                  DESIRED   CURRENT   READY   AGE
    65. replicaset.apps/csi-attacher-5984dd5fb4               3         3         3       49m
    66. replicaset.apps/csi-provisioner-56cf669464            3         3         3       49m
    67. replicaset.apps/csi-resizer-644fd596d5                3         3         3       49m
    68. replicaset.apps/csi-snapshotter-78596646cf            3         3         3       49m
    69. replicaset.apps/longhorn-driver-deployer-5f886cb8cb   1         1         1       49m
    70. replicaset.apps/longhorn-ui-5f766d554b                2         2         2       49m

    查看pod状态

    等待Pod启动：

    一旦存储库创建成功，LongHorn系统将启动一系列的Pod。

    检查Pod的状态：

    1. $ kubectl get pods -n longhorn-system  -o wide
    2. # use
    3. kubectl get pods -n longhorn-system -o wide 
    4.  
    5. kubectl get svc -n longhorn-system

    确认所有的Pod都处于"Running"状态。

    1. # 全部是 Running和状态READY左右两边一致就可以了
    2. [root@master ~]# kubectl get pod \
    3. -n longhorn-system \
    4. -o wide 
    5. NAME                                                  READY   STATUS    RESTARTS       AGE     IP               NODE    NOMINATED NODE   READINESS GATES
    6. csi-attacher-7bf4b7f996-ffhxm                         1/1     Running   0              45m     10.244.166.146   node1   <none>           <none>
    7. csi-attacher-7bf4b7f996-kts4l                         1/1     Running   0              45m     10.244.104.20    node2   <none>           <none>
    8. csi-attacher-7bf4b7f996-vtzdb                         1/1     Running   0              45m     10.244.166.142   node1   <none>           <none>
    9. csi-provisioner-869bdc4b79-bjf8q                      1/1     Running   0              45m     10.244.166.150   node1   <none>           <none>
    10. csi-provisioner-869bdc4b79-m2swk                      1/1     Running   0              45m     10.244.104.18    node2   <none>           <none>
    11. csi-provisioner-869bdc4b79-pmkfq                      1/1     Running   0              45m     10.244.166.143   node1   <none>           <none>
    12. csi-resizer-869fb9dd98-6ndgb                          1/1     Running   0              8m47s   10.244.104.28    node2   <none>           <none>
    13. csi-resizer-869fb9dd98-czvzh                          1/1     Running   0              45m     10.244.104.16    node2   <none>           <none>
    14. csi-resizer-869fb9dd98-pq2p5                          1/1     Running   0              45m     10.244.166.149   node1   <none>           <none>
    15. csi-snapshotter-7d59d56b5c-85cr6                      1/1     Running   0              45m     10.244.104.19    node2   <none>           <none>
    16. csi-snapshotter-7d59d56b5c-dlwjk                      1/1     Running   0              45m     10.244.166.148   node1   <none>           <none>
    17. csi-snapshotter-7d59d56b5c-xsc6s                      1/1     Running   0              45m     10.244.166.147   node1   <none>           <none>
    18. engine-image-ei-f9e7c473-ld6zp                        1/1     Running   0              46m     10.244.104.15    node2   <none>           <none>
    19. engine-image-ei-f9e7c473-qw96n                        1/1     Running   1 (6m5s ago)   8m13s   10.244.166.154   node1   <none>           <none>
    20. instance-manager-e-16be548a213303f54febe8742dc8e307   1/1     Running   0              8m19s   10.244.104.29    node2   <none>           <none>
    21. instance-manager-e-1e8d2b6ac4bdab53558aa36fa56425b5   1/1     Running   0              7m58s   10.244.166.155   node1   <none>           <none>
    22. instance-manager-r-16be548a213303f54febe8742dc8e307   1/1     Running   0              46m     10.244.104.14    node2   <none>           <none>
    23. instance-manager-r-1e8d2b6ac4bdab53558aa36fa56425b5   1/1     Running   0              7m31s   10.244.166.156   node1   <none>           <none>
    24. longhorn-admission-webhook-69979b57c4-rt7rh           1/1     Running   0              52m     10.244.166.136   node1   <none>           <none>
    25. longhorn-admission-webhook-69979b57c4-s2bmd           1/1     Running   0              52m     10.244.104.11    node2   <none>           <none>
    26. longhorn-conversion-webhook-966d775f5-st2ld           1/1     Running   0              52m     10.244.166.135   node1   <none>           <none>
    27. longhorn-conversion-webhook-966d775f5-z5m4h           1/1     Running   0              52m     10.244.104.12    node2   <none>           <none>
    28. longhorn-csi-plugin-gcngb                             3/3     Running   0              45m     10.244.166.145   node1   <none>           <none>
    29. longhorn-csi-plugin-z4pr6                             3/3     Running   0              45m     10.244.104.17    node2   <none>           <none>
    30. longhorn-driver-deployer-5d74696c6-g2p7p              1/1     Running   0              52m     10.244.104.9     node2   <none>           <none>
    31. longhorn-manager-j69pn                                1/1     Running   0              52m     10.244.166.139   node1   <none>           <none>
    32. longhorn-manager-rnzwr                                1/1     Running   0              52m     10.244.104.10    node2   <none>           <none>
    33. longhorn-recovery-backend-6576b4988d-l4lmb            1/1     Running   0              52m     10.244.104.8     node2   <none>           <none>
    34. longhorn-recovery-backend-6576b4988d-v79b4            1/1     Running   0              52m     10.244.166.138   node1   <none>           <none>
    35. longhorn-ui-596d5f6876-ms4dn                          1/1     Running   0              52m     10.244.166.137   node1   <none>           <none>
    36. longhorn-ui-596d5f6876-td7ww                          1/1     Running   0              52m     10.244.104.7     node2   <none>           <none>
    37. [root@master ~]#

    设置svc服务(可选)

    1. ## 安装好后名为longhorn-frontend的svc服务默认是clusterip模式，
    2. ## 除了集群之外的网络是访问不到此服务的，所以要将此svc服务改为nodeport模式
    3. $. kubectl edit svc longhorn-frontend -n longhorn-system
    4. ##type: NodePort    
    5. # 将type的ClusterIP改为 NodePort 即可
    6. # 设置 spec.[ports].nodePort: 32183
    7. # 保存退出
    1. $. kubectl get svc -n longhorn-system
    2.  
    3. # 过滤
    4. [root@master ~]# kubectl get svc -n longhorn-system|grep longhorn-frontend 
    5. longhorn-frontend             ClusterIP   10.106.154.54    <none>        80/TCP      55m
    6. [root@master ~]# 
    1. # 查询刚才变更为 NodePort 暴露的端口
    2. [root@master ~]# kubectl get svc \
    3. -n longhorn-system|grep longhorn-frontend 
    4. longhorn-frontend             NodePort    10.106.154.54            80:32146/TCP   58m
    5. [root@master ~]#

    浏览器访问

    在浏览器访问此端口即可

    1.  http://ip:32183  # 根据自己的k8s宿主机ip地址输入
    2. # rpp
    3. http://192.168.1.190:32183

    使用iscsi类型的volume

    参考: cilium-cew-longhorn

    创建 iscsi volume

    使用 UI 创建新卷，将frontend设置为“iSCSI”，并attached 到集群中主机, 步骤如下:

    步骤1

    步骤2

    步骤3

    步骤4

    验证状态

    1. $. kubectl get lhv test-volume-iscsi -n longhorn-system
    2. NAME                DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE          AGE
    3. test-volume-iscsi   v1            attached   healthy                  2147483648   k8s-infra01   105m
    4.  
    5. # ======================== 参考 ============================ #
    6. # 注: lhv 是 longhorn volume 资源的简称
    7. $ kubectl get lhv test -n longhorn-system
    8. NAME   STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE             AGE
    9. test   attached   healthy      True        21474836480   192.168.20.184   25s

    获取iSCSI volume endpoint

    通过 UI（在“卷详细信息”下）或kubectl, 获取volume的 iSCSI endpoint.

    通过kubectl的方式为:

    1. $. kubectl get lhe -n longhorn-system
    2. NAME                    DATA ENGINE   STATE     NODE          INSTANCEMANAGER                                     IMAGE                                    AGE
    3. test-volume-iscsi-e-0   v1            running   k8s-infra01   instance-manager-1388698f2508ae1055185ec20ce07491   longhornio/longhorn-engine:master-head   8m7s
    4.  
    5. $. kubectl get lhe test-volume-iscsi-e-0 \
    6. -n longhorn-system \
    7. -o jsonpath='{.status.endpoint}'
    8. # 输出
    9. iscsi://10.233.94.197:3260/iqn.2019-10.io.longhorn:test-volume-iscsi/1
    10.  
    11. # use
    12. $. kubectl get lhe sts-iscsi-volume01-e-0 \
    13. -n longhorn-system \
    14. -o jsonpath='{.status.endpoint}'
    15.  
    16. # ======================== 参考 ============================ #
    17. # 注: lhe 是 longhorn endpoint 资源的简称
    18. $ kubectl get lhe -n longhorn-system
    19. NAME              STATE     NODE             INSTANCEMANAGER               IMAGE                               AGE
    20. test-e-b8eb676b   running   192.168.20.184   instance-manager-e-baea466a   longhornio/longhorn-engine:v1.1.2   17m
    21.  
    22. $ kubectl get lhe test-e-b8eb676b -n longhorn-system -o jsonpath='{.status.endpoint}'
    23. iscsi://10.0.2.24:3260/iqn.2019-10.io.longhorn:test/1

    目录挂载卷(方式1,可选)

    通过 iSCSI 启动器（客户端）连接到 Longhorn 卷。

    创建 iSCSI 启动器（客户端）

    Connect iSCSI initiator (client) to Longhorn volume

    将集群外部虚拟机连接到目标集群中的 Longhorn 卷：

    1. # iscsi://10.233.94.197:3260/iqn.2019-10.io.longhorn:test-volume-iscsi/1
    2. # 在客户端机器节点执行
    3. $. iscsiadm --mode discoverydb \
    4. --type sendtargets \
    5. --portal 10.233.94.197 \
    6. --discover 10.233.94.197:3260,1 \
    7. iqn.2019-10.io.longhorn:test-volume-iscsi
    8.  
    9. $. iscsiadm --mode node \
    10. --targetname iqn.2019-10.io.longhorn:test-volume-iscsi \
    11. --portal 10.233.94.197:3260 \
    12. --login
    13.  
    14. $. iscsiadm --mode node
    15. 10.0.2.24:3260,1 iqn.2019-10.io.longhorn:test
    16.  
    17. # ======================== 参考 ============================ #
    18. $ iscsiadm --mode discoverydb --type sendtargets --portal 10.0.2.24 --discover
    19. 10.0.2.24:3260,1 iqn.2019-10.io.longhorn:test
    20.  
    21. $ iscsiadm --mode node --targetname iqn.2019-10.io.longhorn:test --portal 10.0.2.24:3260 --login
    22.  
    23. $ iscsiadm --mode node
    24. 10.0.2.24:3260,1 iqn.2019-10.io.longhorn:test

    验证卷

    该卷可用作 /dev/sdb：

    1. $. lsblk
    2. NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    3. ...
    4. sdc                         8:32   0    2G  0 disk
    5.  
    6. $. journalctl -xn 100 | grep sdc
    7. # 输出
    8. Aug 07 16:52:41 k8s-worker2 kernel: sd 34:0:0:1: [sdc] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
    9. Aug 07 16:52:41 k8s-worker2 kernel: sd 34:0:0:1: [sdc] Write Protect is off
    10. Aug 07 16:52:41 k8s-worker2 kernel: sd 34:0:0:1: [sdc] Mode Sense: 69 00 10 08
    11. Aug 07 16:52:41 k8s-worker2 kernel: sd 34:0:0:1: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
    12. Aug 07 16:52:41 k8s-worker2 kernel: sd 34:0:0:1: [sdc] Attached SCSI disk
    13.  
    14.  
    15. $. lsblk /dev/sdc
    16. NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    17. sdc    8:32   0   2G  0 disk
    18.  
    19. # 接下来挂载该硬盘, 挂载步骤
    20.   # 建立分区
    21.   $. fdisk /dev/sdc
    22.     # 直接默认回车即可
    23.  
    24.   # 查看已有分区: 
    25.   $. fdisk -l
    26.     Device     Boot Start     End Sectors Size Id Type
    27.     /dev/sdc1        2048 4194303 4192256   2G 83 Linux
    28.   
    29.   # 格式化分区, 建立文件系统
    30.   $. mkfs.xfs -f /dev/sdc1
    31.  
    32.   # 挂载到 /test 目录下
    33.   $. mount /dev/sdc1 /test
    34.   # 卸载: umount /dev/sdc1
    35. # ======================== 参考 ============================ #
    36. $ journalctl -xn 100 | grep sdb
    37. Aug 12 09:05:11 longhorn-client kernel: sd 3:0:0:1: [sdb] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
    38. Aug 12 09:05:11 longhorn-client kernel: sd 3:0:0:1: [sdb] Write Protect is off
    39. Aug 12 09:05:11 longhorn-client kernel: sd 3:0:0:1: [sdb] Mode Sense: 69 00 10 08
    40. Aug 12 09:05:11 longhorn-client kernel: sd 3:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
    41. Aug 12 09:05:11 longhorn-client kernel: sd 3:0:0:1: [sdb] Attached SCSI disk
    42.  
    43. $ lsblk /dev/sdb
    44. NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    45. sdb    8:16   0  20G  0 disk

    k8s pod挂载卷(方式2,可选)

    参考: Kubernetes接入iSCSI存储

    首先参考《创建iscsi volume》章节，创建名为test-iscsi-volume的volume.

    创建pv, pvc

    注: 该pvc只能被一个pod挂载

    1. # vi pvc-longhorn-iscsi.yaml
    2. # kubectl apply -f pvc-longhorn-iscsi.yaml
    3. # kubectl delete -f pvc-longhorn-iscsi.yaml
    4. # kubectl get pvc -n rpp-ns
    5. # kubectl delete pvc pvc-longhorn-iscsi -n rpp-ns
    6. # kubectl describe pvc pvc-longhorn-iscsi -n rpp-ns
    7. # iscsi://10.233.94.197:3260/iqn.2019-10.io.longhorn:test-iscsi-volume/1
    8. apiVersion: v1
    9. kind: PersistentVolume
    10. metadata:
    11.   name: pv-longhorn-iscsi
    12. spec:
    13.   capacity:
    14.     storage: 7Gi
    15.   accessModes:
    16.     - ReadWriteOnce
    17.     #- ReadWriteMany
    18.   iscsi:
    19.     targetPortal: 10.233.94.197:3260 # Target IP：Port
    20.     iqn: iqn.2019-10.io.longhorn:test-iscsi-volume # Target IQN
    21.     lun: 1 # 对应到Target LUN号
    22.     fsType: ext4
    23.     #fsType: xfs
    24.     readOnly: false
    25. ---
    26. apiVersion: "v1"
    27. kind: PersistentVolumeClaim
    28. metadata:
    29.   name: pvc-longhorn-iscsi
    30.   namespace: rpp-ns
    31. spec:
    32.   accessModes:
    33.     - "ReadWriteOnce" # iSCSI不支持ReadWriteMany
    34.   resources:
    35.     requests:
    36.       storage: 2Gi
    37.   volumeName: "pv-longhorn-iscsi"
    38.   storageClassName: "" #由于设置了默认的sc，需要强制指定为空，才会使用上面的pv去创建pvc
    39.  
    40. # ======================== 参考 ============================ #
    41. apiVersion: v1
    42. kind: PersistentVolume
    43. metadata:
    44.   name: test
    45. spec:
    46.   capacity:
    47.     storage: 200Gi
    48.   accessModes:
    49.     - ReadWriteOnce
    50.   iscsi:
    51.     targetPortal: 172.16.28.141:3260 #Target IP：Port
    52.     iqn: iqn.2004-12.com.inspur:mcs.as5300g2.node1 # Target IQN
    53.     lun: 0 # 对应到Target LUN号
    54.     fsType: ext4
    55.     readOnly: false
    56. ---
    57. apiVersion: "v1"
    58. kind: PersistentVolumeClaim
    59. metadata:
    60.   name: "test-pvc"
    61. spec:
    62.   accessModes:
    63.     - "ReadWriteOnce" # iSCSI不支持ReadWriteMany
    64.   resources:
    65.     requests:
    66.       storage: "200Gi"
    67.   volumeName: "test"
    68.   storageClassName: "" #由于设置了默认的sc，需要强制指定为空，才会使用test pv去创建pvc
    69.

    创建POD

    1. # vi pod-rpp-longhorn-iscsi.yaml
    2. # kubectl apply -f pod-rpp-longhorn-iscsi.yaml
    3. # kubectl get pod -n rpp-ns
    4.  
    5. # kubectl describe pod pod-rpp-longhorn-iscsi -n rpp-ns
    6. # kubectl exec -it -n rpp-ns pod-rpp-longhorn-iscsi -- sh
    7. # kubectl delete -f pod-rpp-longhorn-iscsi.yaml
    8. apiVersion: v1
    9. kind: Pod
    10. metadata:
    11.   name: pod-rpp-longhorn-iscsi
    12.   namespace: rpp-ns
    13. spec:
    14.   nodeSelector:
    15.     #kubernetes.io/hostname: k8s-infra01  # 必须与 longhorn 存储在同一个node上
    16.     kubernetes.io/hostname: k8s-worker2 
    17.   containers:
    18.   - name: springboot-rpp-longhorn-iscsi  # 容器名
    19.     image: harbor.echo01.tzncloud.com/my-project/spring-boot-docker2:0.0.1-SNAPSHOT
    20.     volumeMounts:
    21.     - name: loghorn-volume
    22.       mountPath: /data_loghorn
    23.     ports:                 # 端口
    24.     - containerPort: 8600  # 容器暴露的端口
    25.       name: business-port
    26.     - containerPort: 8800
    27.       name: actuator-port
    28.   volumes:
    29.   - name: loghorn-volume
    30.     persistentVolumeClaim:
    31.       claimName: pvc-longhorn-iscsi

    k8s sts挂载卷(方式3, use)

    创建 pv

    注: 该pvc只能被一个pod挂载

    1. # vi pv-sts-longhorn-iscsi.yaml
    2. # kubectl apply -f pv-sts-longhorn-iscsi.yaml
    3. # kubectl delete -f pv-sts-longhorn-iscsi.yaml
    4. # kubectl get pv
    5.  
    6. ---
    7. # iscsi://10.233.94.197:3260/iqn.2019-10.io.longhorn:sts-iscsi-volume01/1
    8. apiVersion: v1
    9. kind: PersistentVolume
    10. metadata:
    11.   name: pv-sts-iscsi-volume01
    12. spec:
    13.   storageClassName: sts-manual
    14.   capacity:
    15.     storage: 1Gi
    16.   accessModes:
    17.     - ReadWriteOnce
    18.     #- ReadWriteMany
    19.   iscsi:
    20.     targetPortal: 10.233.94.197:3260 # Target IP：Port
    21.     iqn: iqn.2019-10.io.longhorn:sts-iscsi-volume01 # Target IQN
    22.     lun: 1 # 对应到Target LUN号
    23.     fsType: ext4
    24.     #fsType: xfs
    25.     readOnly: false
    26. ---
    27. apiVersion: v1
    28. kind: PersistentVolume
    29. metadata:
    30.   name: pv-sts-iscsi-volume02
    31. spec:
    32.   storageClassName: sts-manual
    33.   capacity:
    34.     storage: 1Gi
    35.   accessModes:
    36.     - ReadWriteOnce
    37.     #- ReadWriteMany
    38.   iscsi:
    39.     targetPortal: 10.233.94.197:3260 # Target IP：Port
    40.     iqn: iqn.2019-10.io.longhorn:sts-iscsi-volume02 # Target IQN
    41.     lun: 1 # 对应到Target LUN号
    42.     fsType: ext4
    43.     #fsType: xfs
    44.     readOnly: false
    45. ---
    46. apiVersion: v1
    47. kind: PersistentVolume
    48. metadata:
    49.   name: pv-sts-iscsi-volume03
    50. spec:
    51.   storageClassName: sts-manual
    52.   capacity:
    53.     storage: 1Gi
    54.   accessModes:
    55.     - ReadWriteOnce
    56.     #- ReadWriteMany
    57.   iscsi:
    58.     targetPortal: 10.233.94.197:3260 # Target IP：Port
    59.     iqn: iqn.2019-10.io.longhorn:sts-iscsi-volume03 # Target IQN
    60.     lun: 1 # 对应到Target LUN号
    61.     fsType: ext4
    62.     #fsType: xfs
    63.     readOnly: false

    创建statefulset

    1. # vi sts-rpp-longhorn-iscsi.yaml
    2. # kubectl apply -f sts-rpp-longhorn-iscsi.yaml
    3. # kubectl get sts,pod,pvc -n rpp-ns
    4. # kubectl get pod -n rpp-ns
    5.  
    6. # kubectl describe sts-rpp-longhorn-iscsi -n rpp-ns
    7. # kubectl exec -it -n rpp-ns sts-rpp-longhorn-iscsi -- sh
    8. # kubectl delete -f sts-rpp-longhorn-iscsi.yaml
    9. apiVersion: v1
    10. kind: Service
    11. metadata:
    12.   name: springboot-docker2-sts-svc
    13.   namespace: rpp-ns
    14. spec:
    15.   type: NodePort
    16.   selector:     # 标签选择，符合这个标签的 Pod 会作为这个 Service 的 backend。
    17.     app: sts-springboot-docker
    18.   ports:
    19.     - protocol: TCP
    20.       port: 8610        # service暴露在cluster ip上的端口, clusterIP:port 是提供给集群内部客户访问service的入口
    21.       targetPort: 8610  # pod的端口, 从port和nodePort来的流量经过kube-proxy流入到后端pod的targetPort上, 最后进入容器
    22.       nodePort: 30020   # 对外暴露的端口，可以指定, Nodeport端口号范围为: 30000-32767
    23. ---
    24. apiVersion: apps/v1          
    25. kind: StatefulSet               
    26. metadata:                     
    27.   name: sts-springboot-docker
    28.   namespace: rpp-ns
    29.   labels:                     
    30.     app: sts-springboot-docker 
    31. spec:                             
    32.   replicas: 3                      # Pod的副本数
    33.   serviceName: springboot-docker2-sts-svc # 必填参数
    34.   selector:                      
    35.     matchLabels:                     # 标签选择器
    36.       app: sts-springboot-docker   # 选择器，用于匹配Pod的标签，确保Pod属于该sts
    37.   template:                 # Pod的模板信息，根据模板信息来创建Pod   
    38.     metadata:               # Pod的元数据    
    39.       labels:               # Pod的标签，与Selector中的标签匹配以确保Pod属于该sts   
    40.         app: sts-springboot-docker
    41.     spec:                      
    42.       containers:                
    43.       - name: sts-springboot-rpp-longhorn-iscsi  # 容器名
    44.         image: harbor.echo01.tzncloud.com/my-project/spring-boot-docker2:0.0.1-SNAPSHOT
    45.         ports:                 # 端口
    46.         - containerPort: 8600  # 容器暴露的端口
    47.           name: business-port    
    48.         - containerPort: 8800
    49.           name: actuator-port
    50.         volumeMounts:
    51.         - name: pvc-sts-longhorn-iscsi
    52.           mountPath: /data_loghorn
    53.       #volumes:
    54.       #- name: loghorn-volume
    55.       #  persistentVolumeClaim:
    56.       #    claimName: pvc-sts-longhorn-iscsi
    57.   volumeClaimTemplates:
    58.   - metadata:
    59.       name: pvc-sts-longhorn-iscsi
    60.     spec:
    61.       accessModes: [ "ReadWriteOnce" ]
    62.       resources:
    63.         requests:
    64.           storage: 1Gi
    65.       storageClassName: sts-manual   # 这里并没有名为sts-manual的sc, 这里只是表示 PV 是手动配置的，而不是通过 StorageClass 动态配置的

    LongHorn的使用(参考)

    创建LongHorn存储类(SC)：

    接下来，您需要创建一个LongHorn存储类，以便为Kubernetes应用程序提供块存储。您可以将以下内容保存为 longhorn-storageclass.yaml 文件：

    sc-longhorn.yaml(use)

    1. # vi sc-longhorn.yaml
    2. # kubectl apply -f sc-longhorn.yaml
    3. # kubectl delete -f sc-longhorn.yaml
    4. # kubectl get sc
    5. # vi sc-longhorn.yaml
    6. # kubectl apply -f sc-longhorn.yaml
    7. # kubectl describe sc longhorn-sc
    8. apiVersion: storage.k8s.io/v1
    9. kind: StorageClass
    10. metadata:
    11.   name: longhorn-sc
    12. provisioner: driver.longhorn.io
    13. allowVolumeExpansion: true
    14. reclaimPolicy: Delete
    15. parameters:
    16.   numberOfReplicas: "1"
    17.   staleReplicaTimeout: "2880"
    18. ---
    19. apiVersion: storage.k8s.io/v1
    20. kind: StorageClass
    21. metadata:
    22.   name: longhorn-sc
    23.   # namespace: rpp-ns # sc 没有 namespace 属性
    24. provisioner: driver.longhorn.io
    25. allowVolumeExpansion: true
    26. reclaimPolicy: Delete
    27. #parameters:
    28. #  numberOfReplicas: "1"
    29. #  staleReplicaTimeout: "2880"
    30. ---
    31. kind: StorageClass
    32. apiVersion: storage.k8s.io/v1
    33. metadata:
    34.   name: longhorn-test
    35. provisioner: driver.longhorn.io
    36. allowVolumeExpansion: true
    37. reclaimPolicy: Delete
    38. # reclaimPolicy: Retain # 删除pvc时保留pv
    39. volumeBindingMode: Immediate
    40. parameters:
    41.   numberOfReplicas: "3"
    42.   staleReplicaTimeout: "2880"
    43.   fromBackup: ""
    44.   fsType: "ext4"

    官方Storage Class：

    storage-class-parameters

    1. kind: StorageClass
    2. apiVersion: storage.k8s.io/v1
    3. metadata:
    4.   name: longhorn-test
    5. provisioner: driver.longhorn.io
    6. allowVolumeExpansion: true
    7. reclaimPolicy: Delete
    8. # reclaimPolicy: Retain # 删除pvc时保留pv
    9. volumeBindingMode: Immediate
    10. parameters:
    11.   numberOfReplicas: "3"
    12.   staleReplicaTimeout: "2880"
    13.   fromBackup: ""
    14.   fsType: "ext4"
    15. #  mkfsParams: ""
    16. #  migratable: false
    17. #  encrypted: false
    18. #  dataLocality: "disabled"
    19. #  replicaAutoBalance: "ignored"
    20. #  diskSelector: "ssd,fast"
    21. #  nodeSelector: "storage,fast"
    22. #  recurringJobSelector: '[{"name":"snap-group", "isGroup":true},
    23. #                          {"name":"backup", "isGroup":false}]'
    24. #  backingImageName: ""
    25. #  backingImageChecksum: ""
    26. #  backingImageDataSourceType: ""
    27. #  backingImageDataSourceParameters: ""
    28. #  unmapMarkSnapChainRemoved: "ignored"
    29. #  disableRevisionCounter: false
    30. #  replicaSoftAntiAffinity: "ignored"
    31. #  replicaZoneSoftAntiAffinity: "ignored"
    32. #  replicaDiskSoftAntiAffinity: "ignored"
    33. #  nfsOptions: "soft,timeo=150,retrans=3"
    34. #  v1DataEngine: true
    35. #  v2DataEngine: false
    36. #  freezeFSForSnapshot: "ignored"

    创建PVC（PersistentVolumeClaim）：

    pvc-longhorn.yaml (use)

    现在，您可以为应用程序创建一个PersistentVolumeClaim，以便使用LongHorn提供的块存储。您可以将以下内容保存为 longhorn-pvc.yaml 文件：

    1. # vi pvc-longhorn.yaml
    2. # kubectl apply -f pvc-longhorn.yaml --dry-run=client
    3. # kubectl apply -f pvc-longhorn.yaml
    4. # kubectl delete -f pvc-longhorn.yaml
    5. # kubectl get pvc -n rpp-ns
    6. # kubectl delete pvc longhorn-pvc -n rpp-ns
    7. # kubectl describe pvc longhorn-pvc -n rpp-ns
    8. apiVersion: v1
    9. kind: PersistentVolumeClaim
    10. metadata:
    11.   name: longhorn-pvc # 该pvc只能被同一个node上的pod使用
    12.   namespace: rpp-ns
    13. spec:
    14.   storageClassName: longhorn-sc
    15.   accessModes:
    16.   #- ReadWriteOnce
    17.   - ReadWriteMany
    18.   resources:
    19.     requests:
    20.       storage: 10Gi

    pvc-longhorn2.yaml(use)

    1. # vi pvc-longhorn2.yaml
    2. # kubectl apply -f pvc-longhorn2.yaml
    3. # kubectl delete -f pvc-longhorn2.yaml
    4. # kubectl get pvc -n rpp-ns
    5. apiVersion: v1
    6. kind: PersistentVolumeClaim
    7. metadata:
    8.   name: longhorn-pvc2
    9.   namespace: rpp-ns
    10. spec:
    11.   storageClassName: longhorn-sc
    12.   accessModes:
    13.   - ReadWriteOnce
    14.   resources:
    15.     requests:
    16.       storage: 10Gi

    Longhorn PVC with Block Volume Mode - 官方

    对于具有块卷模式的 PVC，Kubelet 在容器内提供块设备时，绝不会尝试更改其权限和所有权。您必须在 pod.spec.securityContext 中设置正确的组 ID，以便 pod 能够读取和写入块设备或以 root 身份运行容器。

    默认情况下，Longhorn 将块设备放入组 ID 6，该组通常与“磁盘”组相关联。因此，使用具有块卷模式的 Longhorn PVC 的 pod 必须在 pod.spec.securityContext 中设置组 ID 6，或以 root 身份运行。例如：

    在 pod.spec.securityContext 中设置组 ID 为 6 的 Pod:
    1. apiVersion: v1
    2. kind: PersistentVolumeClaim
    3. metadata:
    4.   name: longhorn-block-vol
    5. spec:
    6.   accessModes:
    7.     - ReadWriteOnce
    8.   volumeMode: Block  # or Filesystem
    9.   storageClassName: longhorn
    10.   resources:
    11.     requests:
    12.       storage: 2Gi
    13. ---
    14. apiVersion: v1
    15. kind: Pod
    16. metadata:
    17.   name: block-volume-test
    18.   namespace: default
    19. spec:
    20.   securityContext:
    21.     runAsGroup: 1000
    22.     runAsNonRoot: true
    23.     runAsUser: 1000
    24.     supplementalGroups:
    25.     - 6 # sets the group id 6
    26.   containers:
    27.     - name: block-volume-test
    28.       image: ubuntu:20.04
    29.       command: ["sleep", "360000"]
    30.       imagePullPolicy: IfNotPresent
    31.       volumeDevices:
    32.         - devicePath: /dev/longhorn/testblk
    33.           name: block-vol
    34.   volumes:
    35.     - name: block-vol
    36.       persistentVolumeClaim:
    37.         claimName: longhorn-block-vol
    以 root 身份运行的 Pod
    1. apiVersion: v1
    2. kind: PersistentVolumeClaim
    3. metadata:
    4.   name: longhorn-block-vol
    5. spec:
    6.   accessModes:
    7.     - ReadWriteOnce
    8.   volumeMode: Block
    9.   storageClassName: longhorn
    10.   resources:
    11.     requests:
    12.       storage: 2Gi
    13. ---
    14. apiVersion: v1
    15. kind: Pod
    16. metadata:
    17.   name: block-volume-test
    18.   namespace: default
    19. spec:
    20.   containers:
    21.     - name: block-volume-test
    22.       image: ubuntu:20.04
    23.       command: ["sleep", "360000"]
    24.       imagePullPolicy: IfNotPresent
    25.       volumeDevices:
    26.         - devicePath: /dev/longhorn/testblk
    27.           name: block-vol
    28.   volumes:
    29.     - name: block-vol
    30.       persistentVolumeClaim:
    31.         claimName: longhorn-block-vol

    创建POD

    一旦PVC创建成功，您可以将其绑定到您的应用程序中。您可以添加一个示例应用程序Pod，并将挂载PVC作为卷。例如，您可以将以下内容保存为 app-pod.yaml 文件：

    pod-rpp-longhorn.yaml(use)

    1. # vi pod-rpp-longhorn.yaml
    2. # kubectl apply -f pod-rpp-longhorn.yaml --dry-run=client
    3. # kubectl apply -f pod-rpp-longhorn.yaml
    4. # kubectl get pod -n rpp-ns
    5.  
    6. # kubectl describe pod rpp-longhorn-pod -n rpp-ns
    7. # kubectl exec -it -n rpp-ns rpp-longhorn-pod -- sh
    8. # kubectl delete -f rpp-longhorn-pod.yaml
    9. apiVersion: v1
    10. kind: Pod
    11. metadata:
    12.   name: rpp-longhorn-pod
    13.   namespace: rpp-ns
    14. spec:
    15.   nodeSelector:
    16.     kubernetes.io/hostname: k8s-infra01  # 必须与 longhorn 存储在同一个node上
    17.   containers:
    18.   - name: springboot-docker-longhorn  # 容器名
    19.     image: harbor.echo01.tzncloud.com/my-project/spring-boot-docker2:0.0.1-SNAPSHOT
    20.     volumeMounts:
    21.     - name: loghorn-volume
    22.       mountPath: /data_loghorn
    23.     ports:                 # 端口
    24.     - containerPort: 8600  # 容器暴露的端口
    25.       name: business-port
    26.     - containerPort: 8800
    27.       name: actuator-port
    28.   volumes:
    29.   - name: loghorn-volume
    30.     persistentVolumeClaim:
    31.       claimName: longhorn-pvc
    32.       #claimName: test-volume3-pvc
    33. ---
    34. # 参考
    35. # vi app-pod.yaml
    36. # kubectl apply -f app-pod.yaml
    37. apiVersion: v1
    38. kind: Pod
    39. metadata:
    40.   name: app-pod
    41. spec:
    42.   containers:
    43.   - name: app-container
    44.     image: your-app-image
    45.     volumeMounts:
    46.     - name: longhorn-volume
    47.       mountPath: /data
    48.   volumes:
    49.   - name: longhorn-volume
    50.     persistentVolumeClaim:
    51.       claimName: longhorn-volume

    pod-rpp-longhorn2.yaml(use)

    1. # 再创建一个pod, 查看pod中挂载的longhorn目录下是否有第一个pod创建的文件
    2. # vi pod-rpp-longhorn2.yaml
    3. # kubectl apply -f pod-rpp-longhorn2.yaml --dry-run=client
    4. # kubectl apply -f pod-rpp-longhorn2.yaml
    5. # kubectl get pod -n rpp-ns -o wide
    6.  
    7. # kubectl describe pod rpp-longhorn-pod2 -n rpp-ns
    8. # kubectl exec -it -n rpp-ns rpp-longhorn-pod2 -- sh
    9. # kubectl delete -f pod-rpp-longhorn2.yaml
    10. apiVersion: v1
    11. kind: Pod
    12. metadata:
    13.   name: rpp-longhorn-pod2
    14.   namespace: rpp-ns
    15. spec:
    16.   nodeSelector:
    17.     #kubernetes.io/hostname: k8s-worker2
    18.     kubernetes.io/hostname: k8s-infra01
    19.   containers:
    20.   - name: springboot-docker-longhorn2  # 容器名
    21.     image: harbor.echo01.tzncloud.com/my-project/spring-boot-docker2:0.0.1-SNAPSHOT
    22.     volumeMounts:
    23.     - name: loghorn-volume
    24.       mountPath: /data_longhorn
    25.     ports:                 # 端口
    26.     - containerPort: 8600  # 容器暴露的端口
    27.       name: business-port
    28.     - containerPort: 8800
    29.       name: actuator-port
    30.   volumes:
    31.   - name: loghorn-volume
    32.     persistentVolumeClaim:
    33.       #claimName: longhorn-pvc2
    34.       claimName: longhorn-pvc # 如果使用longhorn-pvc, 则与rpp-longhorn-pod共享存储

    PVC 扩容

    查看 StorageClass 是否支持动态扩容：

    如果有参数 allowVolumeExpansion: true 则表示支持。

    在 Longhorn 界面上直接就能进行扩容：

    关于Volume的说明

    删除 Longhorn Volumes

    Deleting Volumes Through Kubernetes

    通过删除使用已配置 Longhorn 卷的 PersistentVolumeClaim 来通过 Kubernetes 删除卷。这将导致 Kubernetes 清理 PersistentVolume，然后删除 Longhorn 中的卷。

    注意：仅当卷由 StorageClass 配置并且 Longhorn 卷的 PersistentVolume 的回收策略设置为删除时，此方法才有效。

    Deleting Volumes Through Longhorn

    • 所有 Longhorn 卷（无论它们是如何创建的）都可以通过 Longhorn UI删除。
    • 要删除单个卷，请转到 UI 中的“卷”页面。在“操作”下拉菜单下，选择“删除”。在删除卷之前，系统会提示您确认。
    • 要同时删除多个卷，您可以在“卷”页面上检查多个卷，然后选择顶部的“删除”。
    • 注意：
      • 如果 Longhorn 检测到卷与 PersistentVolume 或 PersistentVolumeClaim 绑定，则删除卷后这些资源也将被删除。
      • 在继续删除之前，您会在 UI 中收到有关此情况的警告。
      • Longhorn 还会在删除附加卷时向您发出警告，因为它可能正在使用中。

    分离 Longhorn 卷(Detach Longhorn Volumes)

    使用 Longhorn 卷关闭所有 Kubernetes Pod，以便分离卷。
    实现此目的的最简单方法是删除所有工作负载，然后在升级后重新创建它们。
    如果不希望这样做，则可能会暂停某些工作负载。

    在本节中，您将了解如何修改每个工作负载以关闭其 Pod。

    Deployment

    Edit the deployment with kubectl edit deploy/.

    Set .spec.replicas to 0.

    StatefulSet

    使用 kubectl edit statefulset/ 编辑 statefulset。

    将 .spec.replicas 设置为 0。

    DaemonSet

    无法暂停此工作负载。
    使用 kubectl delete ds/ 删除守护程序集。

    Pod

    使用 kubectl delete pod/ 删除 Pod。

    无法暂停未由工作负载控制器管理的 Pod。

    CronJob

    使用 kubectl edit cronjob/ 编辑 cronjob。
    将 .spec.suspend 设置为 true。

    等待当前正在执行的任何作业完成，或通过删除相关 pod 来终止它们。

    Job

    考虑允许单次运行作业完成。

    否则，使用 kubectl delete job/ 删除作业。

    ReplicaSet副本集

    使用 kubectl edit replicaset/ 编辑副本集。
    将 .spec.replicas 设置为 0。

    ReplicationController复制控制器

    使用 kubectl edit rc/ 编辑复制控制器。
    将 .spec.replicas设置为 0。
    等待 Kubernetes 使用的卷完成分离。

    然后从 Longhorn UI 分离所有剩余卷。这些卷很可能是通过 Longhorn UI 或 REST API 在 Kubernetes 之外创建和附加的。

    ReadWriteMany (RWX) Volume

    Longhorn 通过位于share-manager pods中的 NFSv4 服务器公开常规 Longhorn 卷，从而支持 ReadWriteMany (RWX) 卷。

    RWX 卷仅可通过 NFS 挂载访问。默认情况下, Longhorn 使用 NFS 版本 4.1, 并带有 softerr 挂载选项、timeo 值为"600"、retrans 值为"5"。

    使用 Longhorn Volume 作为 iSCSI Target

    Longhorn 支持 iSCSI 目标前端模式(frontend mode)。可以通过任何 iSCSI 客户端（包括 open-iscsi）和虚拟机管理程序（如 KVM）连接到它，只要它与 Longhorn 系统位于同一网络中即可。

    Longhorn CSI 驱动程序不支持 iSCSI 模式。

    即:iscsi类型的volume不支持使用longhorn类型的sc的pvc挂载

    要使用 iSCSI 目标前端模式启动卷，请在创建卷时选择 iSCSI 作为前端。

    连接卷后，您将在端点字段中看到类似以下内容：

    1. # use 
    2. # 查看登录状态
    3. iscsiadm -m session –R
    4.  
    5. # use
    6. iscsi://192.168.1.181:3260/iqn.2014-09.com.rancher:test-volume/1
    7. # 验证
    8. $. iscsiadm --version
    9.  
    10. iscsiadm --mode discoverydb \
    11. --type sendtargets \
    12. --portal 192.168.1.181:3260 --discover
    13.  
    14. # ==================== 参考 =========================
    15. iscsi://10.42.0.21:3260/iqn.2014-09.com.rancher:testvolume/1
    16.  
    17. kubectl get lhe -n longhorn-system
    • IP 和端口为 10.42.0.21:3260。
    • 目标名称为 iqn.2014-09.com.rancher:testvolume。
    • 卷名称为 testvolume。
    • LUN 编号为 1。Longhorn 始终使用 LUN 1。

    上述信息可用于使用 iSCSI 客户端连接到 Longhorn 提供的 iSCSI 目标。

  • 相关阅读:
    Ipad平替电容笔哪个品牌口碑好?apple pencil平替笔排名
    爬虫过程和反爬
    Springboot毕设项目基于springboot的游戏交易网络无忧3op09（java+VUE+Mybatis+Maven+Mysql）
    详解欧拉计划第349题：兰顿的蚂蚁
    Python编程实例-Matplotlib实时数据可视化
    电商系统中的掉单问题
    2023高教社杯全国大学生数学建模竞赛选题建议
    GZ038 物联网应用开发赛题第5套
    PyQt5 QDockWidget
    Dapr在Java中的实践 之 环境准备
  • 原文地址：https://blog.csdn.net/justlpf/article/details/141181258