• StatefulSet 简单实践 Kubernetes

    概述

    在Kubernetes集群中部署MySQL和Mongodb的StatefulSet服务。

    MySQL有官方文档的指引 + 其他网站博客的指引实现

    Mongodb修改operator的Deployment进行简单的实现

    MySQL-StatefulSet

    参考官方文档：运行一个有状态的应用程序 | Kubernetes

    深入剖析Kubernetes学习笔记：StatefulSet-MySQL集群（20） - 活的潇洒80 - 博客园 (cnblogs.com)

    namespace.yaml

    1. apiVersion: v1
    2. kind: Namespace
    3. metadata:
    4.   labels:
    5.     kubernetes.io/metadata.name: mysql
    6.   name: mysql
    7. spec:
    8.   finalizers:
    9.     - kubernetes
    10. status:
    11.   phase: Active

    mysql-class.yaml

    创建动态存储卷：storeageclass。

    存在部署依赖，需要先安装服务：nfs-subdir-external-provisioner

    1. apiVersion: storage.k8s.io/v1
    2. kind: StorageClass
    3. metadata:
    4.   name: nfs-mysql
    5.   namespace: mysql
    6.   annotations:
    7.     storageclass.kubernetes.io/is-default-class: "false"  ## 是否设置为默认的storageclass
    8. provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
    9. #provisioner: nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
    10. parameters:
    11.   archiveOnDelete: "true"
    12.   pathPattern: "${.PVC.namespace}/${.PVC.name}/${.PVC.annotations.nfs.io/storage-path}"
    13.   # pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"
    14.   # pathPattern: "${.PVC.namespace}-${.PVC.name}"

    mysql-pvc.yaml

    参考官方文档：存储类 | Kubernetes

    存在部署依赖，需要先安装服务：nfs-subdir-external-provisioner

    自动创建的，下面data-mysql-0为例子：

    1. apiVersion: v1
    2. kind: PersistentVolumeClaim
    3. metadata:
    4.   annotations:
    5.     pv.kubernetes.io/bind-completed: "yes"
    6.     pv.kubernetes.io/bound-by-controller: "yes"
    7.     volume.beta.kubernetes.io/storage-class: nfs-mysql
    8.     volume.beta.kubernetes.io/storage-provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
    9.   finalizers:
    10.   - kubernetes.io/pvc-protection
    11.   name: data-mysql-0
    12.   namespace: mysql
    13. spec:
    14.   accessModes:
    15.   - ReadWriteMany
    16.   resources:
    17.     requests:
    18.       storage: 10Mi
    19.   storageClassName: nfs-mysql
    20.   volumeMode: Filesystem
    21.

    configmap.yaml

    参考文档：运行一个有状态的应用程序 | Kubernetes

    参考文档：使用StatefulSet搭建MySQL8集群 - 简书 (jianshu.com)

    目前使用的是5.7.43的mysql

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: mysql
    5.   namespace: mysql
    6.   labels:
    7.     app: mysql
    8. data:
    9.   master.cnf: |
    10.     # Apply this config only on the master.
    11.     [client]
    12.     default-character-set=utf8mb4
    13.  
    14.     [mysql]
    15.     default-character-set=utf8mb4
    16.  
    17.     [mysqld]
    18.     # 打开binlog日志
    19.     log-bin
    20.     # binlog_expire_logs_seconds=2592000
    21.     max_connections=10000
    22.     # 在容器里面需要设置下时区
    23.     default-time-zone='+8:00'
    24.     character-set-client-handshake=FALSE
    25.     character-set-server=utf8mb4
    26.     collation-server=utf8mb4_unicode_ci
    27.     init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
    28.   slave.cnf: |
    29.     # Apply this config only on slaves.
    30.     [client]
    31.     default-character-set=utf8mb4
    32.  
    33.     [mysql]
    34.     default-character-set=utf8mb4
    35.  
    36.     [mysqld]
    37.     # 机器设置只读
    38.     super-read-only
    39.     max_connections=10000
    40.     default-time-zone='+8:00'
    41.     character-set-client-handshake=FALSE
    42.     character-set-server=utf8mb4
    43.     collation-server=utf8mb4_unicode_ci
    44.     init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'

    service.yaml

    根据官方文档部署svc，无头服务 + ClusterIP

    1. # 为 StatefulSet 成员提供稳定的 DNS 表项的无头服务（Headless Service）
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: mysql
    6.   namespace: mysql
    7.   labels:
    8.     app: mysql
    9.     app.kubernetes.io/name: mysql
    10. spec:
    11.   ports:
    12.     - name: mysql
    13.       port: 3306
    14.   clusterIP: None
    15.   selector:
    16.     app: mysql
    17. ---
    18. # 用于连接到任一 MySQL 实例执行读操作的客户端服务
    19. # 对于写操作，你必须连接到主服务器：mysql-0.mysql
    20. apiVersion: v1
    21. kind: Service
    22. metadata:
    23.   name: mysql-read
    24.   namespace: mysql
    25.   labels:
    26.     app: mysql
    27.     app.kubernetes.io/name: mysql
    28.     readonly: "true"
    29. spec:
    30.   ports:
    31.     - name: mysql
    32.       port: 3306
    33.   selector:
    34.     app: mysql

    secret.yaml

    官方文档的 MYSQL_ALLOW_EMPTY_PASSWORD 是通过明文的方式传递，需要修复为secret的模式登录，参考文档：Secret | Kubernetes

    创建命令：

    kubectl create secret generic mysql-secret -n mysql  --dry-run=client --from-literal=MYSQL_ALLOW_EMPTY_PASSWORD=1 -o yaml

     结果输出为：

    1. apiVersion: v1
    2. data:
    3.   MYSQL_ALLOW_EMPTY_PASSWORD: MQ==
    4. kind: Secret
    5. metadata:
    6.   creationTimestamp: null
    7.   name: mysql-secret
    8.   namespace: mysql

    使用 Secret 安全地分发凭据 | Kubernetes 配置statefulset

    1. ...
    2.           env:
    3.             - name: MYSQL_ALLOW_EMPTY_PASSWORD
    4.               valueFrom:
    5.                 secretKeyRef:
    6.                   name: mysql-secret
    7.                   key: MYSQL_ALLOW_EMPTY_PASSWORD
    8. ...

    详细参考下面的statefulset.yaml

    StatefulSet.yaml

    修改后的配置文件

    与官方的文档主要差异如下：

    • 配置了镜像拉取策略：imagePullPolicy: Always
    • 密码文件使用secret的环境变量方式传递而非明文：secretKeyRef
    • 使用的是image为MySQL:5.7.43
    • 缩小了副本数：replicas: 2
    • 添加了节点反亲和的策略：nodeAffinity + operator: NotIn
    • 更新了镜像地址： image
    • 使用了storeageclass的方式记录data：storageClassName + 更新了访问模式ReadWriteMany
    1. apiVersion: apps/v1
    2. kind: StatefulSet
    3. metadata:
    4.   name: mysql
    5.   namespace: mysql
    6. spec:
    7.   selector:
    8.     matchLabels:
    9.       app: mysql
    10.       app.kubernetes.io/name: mysql
    11.   serviceName: mysql
    12.   replicas: 2
    13.   template:
    14.     metadata:
    15.       labels:
    16.         app: mysql
    17.         app.kubernetes.io/name: mysql
    18.     spec:
    19.       affinity:
    20.         nodeAffinity:
    21.           requiredDuringSchedulingIgnoredDuringExecution:
    22.             nodeSelectorTerms:
    23.               - matchExpressions:
    24.                   - key: kubernetes.io/hostname
    25.                     operator: NotIn
    26.                     values:
    27.                       - master01
    28.       initContainers:
    29.         - name: init-mysql
    30.           image: mysql:5.7.43
    31.           command:
    32.             - bash
    33.             - "-c"
    34.             - |
    35.               set -ex
    36.               # 基于 Pod 序号生成 MySQL 服务器的 ID。
    37.               [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
    38.               ordinal=${BASH_REMATCH[1]}
    39.               echo [mysqld] > /mnt/conf.d/server-id.cnf
    40.               # 添加偏移量以避免使用 server-id=0 这一保留值。
    41.               echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
    42.               # 将合适的 conf.d 文件从 config-map 复制到 emptyDir。
    43.               if [[ $ordinal -eq 0 ]]; then
    44.                 cp /mnt/config-map/master.cnf /mnt/conf.d/
    45.               else
    46.                 cp /mnt/config-map/slave.cnf /mnt/conf.d/
    47.               fi
    48.           volumeMounts:
    49.             - name: conf
    50.               mountPath: /mnt/conf.d
    51.             - name: config-map
    52.               mountPath: /mnt/config-map
    53.         - name: clone-mysql
    54.           image: reporsitory:5000/xtrabackup:1.0
    55.           imagePullPolicy: Always
    56.           command:
    57.             - bash
    58.             - "-c"
    59.             - |
    60.               set -ex
    61.               # 如果已有数据，则跳过克隆。
    62.               [[ -d /var/lib/mysql/mysql ]] && exit 0
    63.               # 跳过主实例（序号索引 0）的克隆。
    64.               [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
    65.               ordinal=${BASH_REMATCH[1]}
    66.               [[ $ordinal -eq 0 ]] && exit 0
    67.               # 从原来的对等节点克隆数据。
    68.               ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
    69.               # 准备备份。
    70.               xtrabackup --prepare --target-dir=/var/lib/mysql
    71.           volumeMounts:
    72.             - name: data
    73.               mountPath: /var/lib/mysql
    74.               subPath: mysql
    75.             - name: conf
    76.               mountPath: /etc/mysql/conf.d
    77.       containers:
    78.         - name: mysql
    79.           image: mysql:5.7.43
    80.           env:
    81.             - name: MYSQL_ALLOW_EMPTY_PASSWORD
    82.               valueFrom:
    83.                 secretKeyRef:
    84.                   name: mysql-secret
    85.                   key: MYSQL_ALLOW_EMPTY_PASSWORD
    86.           ports:
    87.             - name: mysql
    88.               containerPort: 3306
    89.           volumeMounts:
    90.             - name: data
    91.               mountPath: /var/lib/mysql
    92.               subPath: mysql
    93.             - name: conf
    94.               mountPath: /etc/mysql/conf.d
    95.           resources:
    96.             requests:
    97.               cpu: 500m
    98.               memory: 1Gi
    99.           livenessProbe:
    100.             exec:
    101.               command: ["mysqladmin", "ping"]
    102.             initialDelaySeconds: 30
    103.             periodSeconds: 10
    104.             timeoutSeconds: 5
    105.           readinessProbe:
    106.             exec:
    107.               # 检查我们是否可以通过 TCP 执行查询（skip-networking 是关闭的）。
    108.               command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
    109.             initialDelaySeconds: 5
    110.             periodSeconds: 2
    111.             timeoutSeconds: 1
    112.         - name: xtrabackup
    113.           image: reporsitory:5000/xtrabackup:1.0
    114.           imagePullPolicy: Always
    115.           ports:
    116.             - name: xtrabackup
    117.               containerPort: 3307
    118.           command:
    119.             - bash
    120.             - "-c"
    121.             - |
    122.               set -ex
    123.               cd /var/lib/mysql
    124.  
    125.               # 确定克隆数据的 binlog 位置（如果有的话）。
    126.               if [[ -f xtrabackup_slave_info && "x$(" != "x" ]]; then
    127.                 # XtraBackup 已经生成了部分的 “CHANGE MASTER TO” 查询
    128.                 # 因为我们从一个现有副本进行克隆。(需要删除末尾的分号!)
    129.                 cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
    130.                 # 在这里要忽略 xtrabackup_binlog_info （它是没用的）。
    131.                 rm -f xtrabackup_slave_info xtrabackup_binlog_info
    132.               elif [[ -f xtrabackup_binlog_info ]]; then
    133.                 # 我们直接从主实例进行克隆。解析 binlog 位置。
    134.                 [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
    135.                 rm -f xtrabackup_binlog_info xtrabackup_slave_info
    136.                 echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
    137.                       MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
    138.               fi
    139.  
    140.               # 检查我们是否需要通过启动复制来完成克隆。
    141.               if [[ -f change_master_to.sql.in ]]; then
    142.                 echo "Waiting for mysqld to be ready (accepting connections)"
    143.                 until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
    144.  
    145.                 echo "Initializing replication from clone position"
    146.                 mysql -h 127.0.0.1 \
    147.                       -e "$(, \
    148.                               MASTER_HOST='mysql-0.mysql', \
    149.                               MASTER_USER='root', \
    150.                               MASTER_PASSWORD='', \
    151.                               MASTER_CONNECT_RETRY=10; \
    152.                             START SLAVE;" || exit 1
    153.                 # 如果容器重新启动，最多尝试一次。
    154.                 mv change_master_to.sql.in change_master_to.sql.orig
    155.               fi
    156.  
    157.               # 当对等点请求时，启动服务器发送备份。
    158.               exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
    159.                 "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
    160.           volumeMounts:
    161.             - name: data
    162.               mountPath: /var/lib/mysql
    163.               subPath: mysql
    164.             - name: conf
    165.               mountPath: /etc/mysql/conf.d
    166.           resources:
    167.             requests:
    168.               cpu: 100m
    169.               memory: 100Mi
    170.       volumes:
    171.         - name: conf
    172.           emptyDir: {}
    173.         - name: config-map
    174.           configMap:
    175.             name: mysql
    176.   volumeClaimTemplates:
    177.     - metadata:
    178.         name: data
    179.       spec:
    180.         storageClassName: nfs-mysql
    181.         accessModes: ["ReadWriteMany"]
    182.         resources:
    183.           requests:
    184.             storage: 10Mi

    Error: pvc的绑定错误

    使用默认的官方的yaml文件会发生报错，需要配置PV-Claim:

    一直报错：

    修改配置PV-Claim。yaml文件：

    这里应该不用修改PVC，因为pvc是自动创建，应该是修改statefulset中的storageClassName: nfs-mysql 实现修复。

    Error: pause:3.6 镜像无法下载

    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox i                                     mage "registry.aliyuncs.com/google_containers/pause:3.6": failed to pull image "registry.aliyuncs.com/google_containers/pause:3.6": failed to pull and unpack imag                                     e "registry.aliyuncs.com/google_containers/pause:3.6": failed to resolve reference "registry.aliyuncs.com/google_containers/pause:3.6": failed to do request: Head                                      "https://registry.aliyuncs.com/v2/google_containers/pause/manifests/3.6": dial tcp: lookup registry.aliyuncs.com on [fe80::10%ens34]:53: read udp [fe80::d21d:3ae                                     :ea99:27c6%ens34]:51322->[fe80::10%ens34]:53: i/o timeout

    因为被调度到master01节点，该节点没有pause:3.6镜像，所以启动失败，添加反亲和策略：

    将 Pod 指派给节点 | Kubernetes

    解决办法如下：

    Error: xtrabackup:1.0镜像下载失败

    更换以下镜像源：

    使用StatefulSet搭建MySQL8集群 - 简书 (jianshu.com)

    mzmuer/xtrabackup:1.0

    又或者docker search:

    但仍然是失败的，核心应该是修改： /etc/containerd/config.toml 。

    因为目前集群使用的是containerd，镜像仓库是docker的registry，需要修改对应的配置：

    1. 在仓库主机中将镜像ist0ne/xtrabackup下载下来
    2. 使用docker tag 更新标签
    3. 从新推送到自己的镜像仓库中

    Error: init-container 启动失败

    官方文档使用的是：primary.cnf + replica.cnf

    其他参考文档使用的是：master.cnf + slave.cnf

    我参考的是国内其他参考文档的内容，所以要进行修改

    Error:The node was low on resource: ephemeral-storage

    临时存储空间不足，下载不了镜像

    当时master02 --> master03 --> node04 --> node05 顺序集群自动逐步尝试部署。

    Error: ErrImagePull

    添加镜像拉取策略：imagePullPolicy: Always

    修改containerd的配置文件，一般为：/etc/containerd/config.toml

    随后重启服务： systemctl restart containerd

    因为部署的是docker的镜像仓库：相关的daemon.json如图：

    修复后：

    Error: unknown variable 'binlog_expire_logs_seconds=2592000'

    非官方的参考文档用的是mysql:8的镜像，其中配置文件写binlog_expire_logs_seconds=2592000

    目前，我自己使用的镜像是MySQL:5.7.43的镜像，这里可能存在配置参数不支持的问题

    修改配置文件：kubectl edit configmap mysql -n mysql

    Error: pvc 一直处于pending状态

    相关nfs服务已经处于资源吃紧的边缘：

    缩减资源的使用，statefulset的副本需要缩小，由原来的3个缩小为2个。

    成功结果：

    验证：

    参考文档

    深入剖析Kubernetes学习笔记：StatefulSet-MySQL集群（20） - 活的潇洒80 - 博客园 (cnblogs.com)

    kubernetes【statefulset搭建MySQL集群】 - 掘金 (juejin.cn)

    使用pv + nfs服务器的案例：k8s-使用statefulset部署mysql主从集群5.7 - gg888666 - 博客园 (cnblogs.com)

    使用storageclass + PV的案例：K8S 部署 Statefulset mysql-腾讯云开发者社区-腾讯云 (tencent.com)

    有状态的节点控制器 -- StatefulSet 及其网络状态 (qq.com)

    有状态的节点控制器 StatefulSet 的存储状态 (qq.com)

    实战 Kubernetes StatefulSet -- MySQL 主从集群搭建-腾讯云开发者社区-腾讯云 (tencent.com)

    Kubernetes：使用StatefulSet搭建MySQL集群（一主多从） - Cooper_73 - 博客园 (cnblogs.com)

    Kubernetes：使用StatefulSet搭建MySQL集群（一主多从） - Cooper_73 - 博客园 (cnblogs.com)

    Mongodb-StatefulSet

    文件来源

    Install and Configure the Kubernetes Operator — MongoDB Kubernetes Operator upcoming

    Install the MongoDB Enterprise Kubernetes Operator — MongoDB Kubernetes Operator upcoming

    mongodb-enterprise-kubernetes/mongodb-enterprise.yaml at master · mongodb/mongodb-enterprise-kubernetes (github.com)

    crd.yaml

    mongodb-enterprise-kubernetes/crds.yaml at master · mongodb/mongodb-enterprise-kubernetes (github.com)

    来源于官网文件，使用kubectl apply -f 直接部署 

    mongodb-enterprise.yaml

    参考的视频：statefulset 学亮-哔哩哔哩_Bilibili

    将官方网站的deployment文件进行了修改，主要是：

    • 增加了创建namespace的yaml部分
    • 修改了Deployment为StatefulSet
    • 增加了无头服务Service
    1. ---
    2. apiVersion: v1
    3. kind: Namespace
    4. metadata:
    5.   labels:
    6.     kubernetes.io/metadata.name: mongodb
    7.   name: mongodb
    8. spec:
    9.   finalizers:
    10.   - kubernetes
    11. status:
    12.   phase: Active
    13. ---
    14. # Source: enterprise-operator/templates/operator-roles.yaml
    15. apiVersion: v1
    16. kind: ServiceAccount
    17. metadata:
    18.   name: mongodb-enterprise-operator
    19.   namespace: mongodb
    20. ---
    21. # Source: enterprise-operator/templates/operator-roles.yaml
    22. kind: ClusterRole
    23. apiVersion: rbac.authorization.k8s.io/v1
    24. metadata:
    25.   name: mongodb-enterprise-operator-mongodb-webhook
    26. rules:
    27.   - apiGroups:
    28.       - "admissionregistration.k8s.io"
    29.     resources:
    30.       - validatingwebhookconfigurations
    31.     verbs:
    32.       - get
    33.       - create
    34.       - update
    35.       - delete
    36.   - apiGroups:
    37.       - ""
    38.     resources:
    39.       - services
    40.     verbs:
    41.       - get
    42.       - list
    43.       - watch
    44.       - create
    45.       - update
    46.       - delete
    47. ---
    48. # Source: enterprise-operator/templates/operator-roles.yaml
    49. kind: ClusterRoleBinding
    50. apiVersion: rbac.authorization.k8s.io/v1
    51. metadata:
    52.   name: mongodb-enterprise-operator-mongodb-webhook-binding
    53. roleRef:
    54.   apiGroup: rbac.authorization.k8s.io
    55.   kind: ClusterRole
    56.   name: mongodb-enterprise-operator-mongodb-webhook
    57. subjects:
    58.   - kind: ServiceAccount
    59.     name: mongodb-enterprise-operator
    60.     namespace: mongodb
    61. ---
    62. # Source: enterprise-operator/templates/operator-roles.yaml
    63. kind: Role
    64. apiVersion: rbac.authorization.k8s.io/v1
    65. metadata:
    66.   name: mongodb-enterprise-operator
    67.   namespace: mongodb
    68. rules:
    69.   - apiGroups:
    70.       - ""
    71.     resources:
    72.       - services
    73.     verbs:
    74.       - get
    75.       - list
    76.       - watch
    77.       - create
    78.       - update
    79.       - delete
    80.   - apiGroups:
    81.       - ""
    82.     resources:
    83.       - secrets
    84.       - configmaps
    85.     verbs:
    86.       - get
    87.       - list
    88.       - create
    89.       - update
    90.       - delete
    91.       - watch
    92.   - apiGroups:
    93.       - apps
    94.     resources:
    95.       - statefulsets
    96.     verbs:
    97.       - create
    98.       - get
    99.       - list
    100.       - watch
    101.       - delete
    102.       - update
    103.   - apiGroups:
    104.       - ""
    105.     resources:
    106.       - pods
    107.     verbs:
    108.       - get
    109.       - list
    110.       - watch
    111.       - delete
    112.       - deletecollection
    113.   - apiGroups:
    114.       - mongodb.com
    115.     verbs:
    116.       - "*"
    117.     resources:
    118.       - mongodb
    119.       - mongodb/finalizers
    120.       - mongodbusers
    121.       - opsmanagers
    122.       - opsmanagers/finalizers
    123.       - mongodbmulticluster
    124.       - mongodbmulticluster/finalizers
    125.       - mongodb/status
    126.       - mongodbusers/status
    127.       - opsmanagers/status
    128.       - mongodbmulticluster/status
    129. ---
    130. # Source: enterprise-operator/templates/operator-roles.yaml
    131. kind: RoleBinding
    132. apiVersion: rbac.authorization.k8s.io/v1
    133. metadata:
    134.   name: mongodb-enterprise-operator
    135.   namespace: mongodb
    136. roleRef:
    137.   apiGroup: rbac.authorization.k8s.io
    138.   kind: Role
    139.   name: mongodb-enterprise-operator
    140. subjects:
    141.   - kind: ServiceAccount
    142.     name: mongodb-enterprise-operator
    143.     namespace: mongodb
    144.  
    145. # This ClusterRoleBinding is necessary in order to use validating
    146. # webhooks—these will prevent you from applying a variety of invalid resource
    147. # definitions. The validating webhooks are optional so this can be removed if
    148. # necessary.
    149. ---
    150. # Source: enterprise-operator/templates/database-roles.yaml
    151. apiVersion: v1
    152. kind: ServiceAccount
    153. metadata:
    154.   name: mongodb-enterprise-appdb
    155.   namespace: mongodb
    156. ---
    157. # Source: enterprise-operator/templates/database-roles.yaml
    158. apiVersion: v1
    159. kind: ServiceAccount
    160. metadata:
    161.   name: mongodb-enterprise-database-pods
    162.   namespace: mongodb
    163. ---
    164. # Source: enterprise-operator/templates/database-roles.yaml
    165. apiVersion: v1
    166. kind: ServiceAccount
    167. metadata:
    168.   name: mongodb-enterprise-ops-manager
    169.   namespace: mongodb
    170. ---
    171. # Source: enterprise-operator/templates/database-roles.yaml
    172. kind: Role
    173. apiVersion: rbac.authorization.k8s.io/v1
    174. metadata:
    175.   name: mongodb-enterprise-appdb
    176.   namespace: mongodb
    177. rules:
    178.   - apiGroups:
    179.       - ""
    180.     resources:
    181.       - secrets
    182.     verbs:
    183.       - get
    184.   - apiGroups:
    185.       - ""
    186.     resources:
    187.       - pods
    188.     verbs:
    189.       - patch
    190.       - delete
    191.       - get
    192. ---
    193. # Source: enterprise-operator/templates/database-roles.yaml
    194. kind: RoleBinding
    195. apiVersion: rbac.authorization.k8s.io/v1
    196. metadata:
    197.   name: mongodb-enterprise-appdb
    198.   namespace: mongodb
    199. roleRef:
    200.   apiGroup: rbac.authorization.k8s.io
    201.   kind: Role
    202.   name: mongodb-enterprise-appdb
    203. subjects:
    204.   - kind: ServiceAccount
    205.     name: mongodb-enterprise-appdb
    206.     namespace: mongodb
    207. ---
    208. # Source: enterprise-operator/templates/operator.yaml
    209. apiVersion: apps/v1
    210. kind: StatefulSet
    211. metadata:
    212.   name: mongodb-enterprise-operator
    213.   namespace: mongodb
    214. spec:
    215.   serviceName: mongodb
    216.   replicas: 3
    217.   selector:
    218.     matchLabels:
    219.       app.kubernetes.io/component: controller
    220.       app.kubernetes.io/name: mongodb-enterprise-operator
    221.       app.kubernetes.io/instance: mongodb-enterprise-operator
    222.   template:
    223.     metadata:
    224.       labels:
    225.         app.kubernetes.io/component: controller
    226.         app.kubernetes.io/name: mongodb-enterprise-operator
    227.         app.kubernetes.io/instance: mongodb-enterprise-operator
    228.     spec:
    229.       serviceAccountName: mongodb-enterprise-operator
    230.       securityContext:
    231.         runAsNonRoot: true
    232.         runAsUser: 2000
    233.       containers:
    234.         - name: mongodb-enterprise-operator
    235.           image: "quay.io/mongodb/mongodb-enterprise-operator-ubi:1.22.0"
    236.           imagePullPolicy: Always
    237.           args:
    238.             - -watch-resource=mongodb
    239.             - -watch-resource=opsmanagers
    240.             - -watch-resource=mongodbusers
    241.           command:
    242.             - /usr/local/bin/mongodb-enterprise-operator
    243.           resources:
    244.             limits:
    245.               cpu: 1100m
    246.               memory: 1Gi
    247.             requests:
    248.               cpu: 500m
    249.               memory: 200Mi
    250.           env:
    251.             - name: OPERATOR_ENV
    252.               value: prod
    253.             - name: WATCH_NAMESPACE
    254.               valueFrom:
    255.                 fieldRef:
    256.                   fieldPath: metadata.namespace
    257.             - name: NAMESPACE
    258.               valueFrom:
    259.                 fieldRef:
    260.                   fieldPath: metadata.namespace
    261.             - name: CLUSTER_CLIENT_TIMEOUT
    262.               value: "10"
    263.             - name: IMAGE_PULL_POLICY
    264.               value: Always
    265.             # Database
    266.             - name: MONGODB_ENTERPRISE_DATABASE_IMAGE
    267.               value: quay.io/mongodb/mongodb-enterprise-database-ubi
    268.             - name: INIT_DATABASE_IMAGE_REPOSITORY
    269.               value: quay.io/mongodb/mongodb-enterprise-init-database-ubi
    270.             - name: INIT_DATABASE_VERSION
    271.               value: 1.0.19
    272.             - name: DATABASE_VERSION
    273.               value: 2.0.2
    274.             # Ops Manager
    275.             - name: OPS_MANAGER_IMAGE_REPOSITORY
    276.               value: quay.io/mongodb/mongodb-enterprise-ops-manager-ubi
    277.             - name: INIT_OPS_MANAGER_IMAGE_REPOSITORY
    278.               value: quay.io/mongodb/mongodb-enterprise-init-ops-manager-ubi
    279.             - name: INIT_OPS_MANAGER_VERSION
    280.               value: 1.0.12
    281.             # AppDB
    282.             - name: INIT_APPDB_IMAGE_REPOSITORY
    283.               value: quay.io/mongodb/mongodb-enterprise-init-appdb-ubi
    284.             - name: INIT_APPDB_VERSION
    285.               value: 1.0.18
    286.             - name: OPS_MANAGER_IMAGE_PULL_POLICY
    287.               value: Always
    288.             - name: AGENT_IMAGE
    289.               value: "quay.io/mongodb/mongodb-agent-ubi:12.0.25.7724-1"
    290.             - name: MONGODB_IMAGE
    291.               value: mongodb-enterprise-server
    292.             - name: MONGODB_REPO_URL
    293.               value: quay.io/mongodb
    294.             - name: MDB_IMAGE_TYPE
    295.               value: ubi8
    296.             - name: PERFORM_FAILOVER
    297.               value: "true"
    298. ---
    299. apiVersion: v1
    300. kind: Service
    301. metadata:
    302.   name: mongodb
    303.   namespace: mongodb
    304. spec:
    305.   selector:
    306.     app.kubernetes.io/name: mongodb-enterprise-operator
    307.   type: ClusterIP
    308.   clusterIP: None
    309.   ports:
    310.     - port: 27017
    311.       targetPort: 27017

    成功部署的结果：

    部署 Ops Manager Resource

    secret.yaml

    1. kubectl create secret generic ops-manager-admin-secret \
    2. --from-literal=Username="test"  --from-literal=Password="test" \
    3. --from-literal=FirstName="test" --from-literal=LastName="test" \
    4. -n mongodb -o yaml --dry-run=client

    storageclass.yaml

    1. apiVersion: storage.k8s.io/v1
    2. kind: StorageClass
    3. metadata:
    4.   name: nfs-mongodb
    5.   namespace: mongodb
    6.   annotations:
    7.     storageclass.kubernetes.io/is-default-class: "false" ## 是否设置为默认的storageclass
    8. provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
    9. #provisioner: nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
    10. parameters:
    11.   archiveOnDelete: "true"
    12.   pathPattern: "${.PVC.namespace}/${.PVC.name}/${.PVC.annotations.nfs.io/storage-path}"
    13.   # pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"
    14.   # pathPattern: "${.PVC.namespace}-${.PVC.name}"

    ops-manager.yaml

    依赖于crd.yaml的成功创建才能执行该步骤

    Error: version + members 错误

    Error： 修改旧版文件，部署不可行

    部署的时候参考第三方旧版文件部署：Kubernetes部署MongoDB集群(一)：安装MongoDB Ops Manager | Finisky Garden

    最终没有成功

    1. apiVersion: mongodb.com/v1
    2. kind: MongoDBOpsManager
    3. metadata:
    4.   name: ops-manager
    5.   namespace: mongodb
    6. spec:
    7.   # the version of Ops Manager distro to use
    8.   # 4.2.4 已经不再支持
    9.   version: 5.0.13    
    10.  
    11.   # the name of the secret containing admin user credentials.
    12.   adminCredentials: ops-manager-admin-secret
    13.  
    14.   externalConnectivity:
    15.     type: NodePort
    16.  
    17.   # the Replica Set backing Ops Manager.
    18.   # appDB has the SCRAM-SHA authentication mode always enabled
    19.   applicationDatabase:
    20.     members: 3
    21.     version: "4.4.0-ubi8"

    查看状态：一直pending

    1. kubectl api-resources | grep mongodb
    2.  
    3. kubectl get opsmanagers -n mongodb
    4.  
    5. kubectl describe opsmanagers.mongodb.com ops-manager      -n mongodb
    6.  
    7. kubectl get all -n mongodb

    核心问题是没有持久卷可以被绑定：

    根据官方文档，

    第一种是PVC的模式部署

    Configure File System Backup Store with Kubernetes Operator — MongoDB Kubernetes Operator upcoming

     第二种是storageclass模式部署

    mongodb-enterprise-kubernetes/samples/ops-manager/ops-manager-local-mode.yaml at master · mongodb/mongodb-enterprise-kubernetes · GitHub根据github仓库选合适的yaml文件部署：ops-manager-local-mode.yaml，修改后部署，该版本使用的镜像较大，最终也没部署下来

    1. apiVersion: mongodb.com/v1
    2. kind: MongoDBOpsManager
    3. metadata:
    4.   name: ops-manager-local
    5.   namespace: mongodb
    6. spec:
    7.   replicas: 1
    8.   version: 5.0.5
    9.   adminCredentials: ops-manager-admin-secret
    10.   configuration:
    11.     # this enables local mode in Ops Manager
    12.     automation.versions.source: local
    13.  
    14.   statefulSet:
    15.     spec:
    16.       # the Persistent Volume Claim will be created for each Ops Manager Pod
    17.       volumeClaimTemplates:
    18.         - metadata:
    19.             name: mongodb-versions
    20.           spec:
    21.             # 定义storageClass
    22.             storageClassName: nfs-mongodb
    23.             # 修改模式为many
    24.             accessModes: ["ReadWriteMany"]
    25.             resources:
    26.               requests:
    27.                 storage: 20G
    28.       template:
    29.         spec:
    30.           containers:
    31.             - name: mongodb-ops-manager
    32.               volumeMounts:
    33.                 - name: mongodb-versions
    34.                   # this is the directory in each Pod where all MongoDB
    35.                   # archives must be put
    36.                   mountPath: /mongodb-ops-manager/mongodb-releases
    37.  
    38.   backup:
    39.     enabled: false
    40.  
    41.   applicationDatabase:
    42.     version: "4.4.11-ent"
    43.     members: 3

    修改PVC.yaml

    一直处于pending状态，故手动对其进行修改：

    手动修改，会提示失败，删除现有的PVC，重新部署/tmp/下的pvc

    1. apiVersion: v1
    2. kind: PersistentVolumeClaim
    3. metadata:
    4.   labels:
    5.     app: ops-manager-local-db-svc
    6.     controller: mongodb-enterprise-operator
    7.     pod-anti-affinity: ops-manager-local-db
    8.   name: data-ops-manager-local-db-0
    9.   namespace: mongodb
    10. spec:
    11. # 新增storageclass
    12.   storageClassName: nfs-mongodb
    13. # 修改访问模式
    14.   accessModes:
    15.   - ReadWriteMany
    16.   resources:
    17.     requests:
    18.       storage: 16G
    19.   volumeMode: Filesystem

    配置节点驱逐：

    配置完成后，删除pod，重新调度部署

    修改镜像配置：

    因为电脑配置较低，需要在镜像服务器将相关镜像推送到个人docker镜像中

    修改statefulset.yaml文件，方便镜像拉取部署。

    修改containerd的启动配置

    Failed to pull image "reporsitory:5000/mongodb-enterprise-init-appdb-ubi:1.0.18": rpc error: code = Unknown desc = f00/mongodb-enterprise-init-appdb-ubi:1.0.18": failed to resolve reference "reporsitory:5000/mongodb-enterprise-init-appdb-ubi:1.0.18": failed to do request: Head "https://reporsitoanifests/1.0.18": http: server gave HTTP response to HTTPS client

    需要新增加为：vim /etc/containerd/config.toml

    随后需要重启服务

    参考文档

    Install the MongoDB Enterprise Kubernetes Operator — MongoDB Kubernetes Operator upcoming

    Deploy an Ops Manager Resource — MongoDB Kubernetes Operator upcoming

    部署Ops Manager资源_MonogDB 中文网 (mongodb.net.cn)

    Kubernetes部署MongoDB集群(一)：安装MongoDB Ops Manager | Finisky Garden

  • 相关阅读:
    Kubernetes Node Not Ready Error
    QT中使用QProcess执行命令，实时获取数据，例如进度条
    (附源码)计算机毕业设计ssm《Java EE开发技术》课程学习网站
    VR云游带你玩转智慧文旅，解决景区营销痛点
    wininet,winhttp,xmlhttprequest,各版本区别 《转》
    网络调试工具编程实现
    Wow: 基于 DDD、EventSourcing 的现代响应式 CQRS 架构微服务开发框架
    Allegro如何使用快捷键快速切换层面操作指导
    channel 进阶
    基于大模型做txt文档拆分的方法
  • 原文地址：https://blog.csdn.net/m0_59267075/article/details/133255012