-
Kubernetes 基于 helm 部署高可用 harbor
Kubernetes 基于 helm 部署高可用 harbor
先决条件
- 高可用入口控制器（Harbor 不管理外部端点） – nodeport方式
- 高可用PostgreSQL 9.6+（Harbor不处理数据库HA的部署） --PostreSQL operator 14
- 高可用Redis（Harbor不处理Redis的HA部署） – redis sentinel
- 可以跨节点或外部对象存储共享的 PVC – minio
存储层方面，需要提供高可用的 PostgreSQL、Redis 集群用于应用数据和 PVC 或对象存储用于存储镜像和helm图表。
1、准备 kubernetes 集群
推荐使用 sealos 快速部署kubernetes集群，准备4个可调度的节点。
root@node01:~# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node01 Ready control-plane 5d12h v1.24.3 192.168.72.50 <none> Ubuntu 22.04 LTS 5.15.0-27-generic containerd://1.6.2 node02 Ready <none> 5d12h v1.24.3 192.168.72.51 <none> Ubuntu 22.04 LTS 5.15.0-27-generic containerd://1.6.2 node03 Ready <none> 5d12h v1.24.3 192.168.72.52 <none> Ubuntu 22.04 LTS 5.15.0-27-generic containerd://1.6.2 node04 Ready <none> 5d12h v1.24.3 192.168.72.53 <none> Ubuntu 22.04 LTS 5.15.0-27-generic containerd://1.6.2- 1
- 2
- 3
- 4
- 5
- 6
2、部署持久化存储
依赖持久化存储的组件：
- redis
- postgresql
- minio
- harbor trivy组件
这里作为测试，使用openebs local-hostpath作为后端存储，实际环境可选择rook、longhorn、nfs或对接外部ceph方式。
helm repo add openebs https://openebs.github.io/charts helm install openebs openebs/openebs -n openebs --create-namespace- 1
- 2
查看存储状态
root@node01:~# kubectl -n openebs get pods NAME READY STATUS RESTARTS AGE openebs-localpv-provisioner-7b7b4c7b7d-87r2n 1/1 Running 4 (146m ago) 5d13h openebs-ndm-cluster-exporter-54cf95c4f7-kzmd5 1/1 Running 0 5d13h openebs-ndm-node-exporter-7zg5r 1/1 Running 0 5d13h openebs-ndm-node-exporter-cqr8f 1/1 Running 0 5d13h openebs-ndm-node-exporter-npt8z 1/1 Running 0 5d13h openebs-ndm-node-exporter-st8vs 1/1 Running 0 5d13h openebs-ndm-operator-6566d67cf6-8l2q8 1/1 Running 0 5d13h- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
确认存在默认存储类，后续操作依赖该存储类
root@node01:~# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 5d13h- 1
- 2
- 3
3、部署 Redis Sentinel 集群
部署redis operator
helm repo add redis-operator https://spotahome.github.io/redis-operator helm install redis-operator redis-operator/redis-operator -n redis --create-namespace \ --set service.type=NodePort- 1
- 2
- 3
- 4
部署redis sentinel 集群
kubectl create -f https://raw.githubusercontent.com/spotahome/redis-operator/master/example/redisfailover/basic.yaml- 1
验证集群状态
root@node01:~# kubectl -n redis get pods NAME READY STATUS RESTARTS AGE redis-operator-5cb98dbf66-gnfr6 1/1 Running 0 3d23h rfr-redisfailover-0 1/1 Running 0 3d23h rfr-redisfailover-1 1/1 Running 0 3d23h rfr-redisfailover-2 1/1 Running 0 3d23h rfs-redisfailover-69c5c99854-4zr2l 1/1 Running 0 3d23h rfs-redisfailover-69c5c99854-ll8tr 1/1 Running 0 3d23h rfs-redisfailover-69c5c99854-m2h9r 1/1 Running 0 3d23h- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
获取redis集群内部service地址
root@node01:~# kubectl -n redis get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redis-operator NodePort 10.96.3.149 <none> 9710:30787/TCP 3d23h rfs-redisfailover ClusterIP 10.96.1.46 <none> 26379/TCP 3d23h- 1
- 2
- 3
- 4
4、部署 PostgreSQL集群
部署postgreSQL operator
helm repo add postgres-operator-charts https://opensource.zalando.com/postgres-operator/charts/postgres-operator helm upgrade -i postgres-operator postgres-operator-charts/postgres-operator -n postgres --create-namespace- 1
- 2
- 3
部署 postgresql ui
helm upgrade -i postgres-operator-ui postgres-operator-ui-charts/postgres-operator-ui \ -n postgres --create-namespace --set service.type=NodePort- 1
- 2
下载 postgresql 集群部署文件
wget https://raw.githubusercontent.com/zalando/postgres-operator/v1.8.2/manifests/minimal-postgres-manifest.yaml- 1
修改 postgresql 集群部署文件，创建 harbor用户，并为 harbor 创建需要的3个数据库：registry、notary_server、notary_signer：
$ cat minimal-postgres-manifest.yaml apiVersion: "acid.zalan.do/v1" kind: postgresql metadata: name: acid-minimal-cluster namespace: postgres spec: teamId: "acid" volume: size: 1Gi numberOfInstances: 2 users: harbor: - superuser - createdb databases: registry: harbor notary_server: harbor notary_signer: harbor preparedDatabases: registry: {} notary_server: {} notary_signer: {} postgresql: version: "14"- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
部署 postgresql 集群
kubectl apply -f minimal-postgres-manifest.yaml- 1
确认集群运行状态
root@node01:~# kubectl -n postgres get pods NAME READY STATUS RESTARTS AGE acid-minimal-cluster-0 1/1 Running 0 5d8h acid-minimal-cluster-1 1/1 Running 0 5d3h postgres-operator-f5c55b6f9-w2f4k 1/1 Running 0 5d4h postgres-operator-ui-5cfff55c65-6c4mk 1/1 Running 0 5d12h- 1
- 2
- 3
- 4
- 5
- 6
获取service连接信息
root@node01:~# kubectl -n postgres get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE acid-minimal-cluster ClusterIP 10.96.3.85 <none> 5432/TCP 5d8h acid-minimal-cluster-config ClusterIP None <none> <none> 5d8h acid-minimal-cluster-repl ClusterIP 10.96.1.103 <none> 5432/TCP 5d8h postgres-operator ClusterIP 10.96.3.114 <none> 8080/TCP 5d12h postgres-operator-ui NodePort 10.96.2.128 <none> 80:31617/TCP 5d12h- 1
- 2
- 3
- 4
- 5
- 6
- 7
获取数据库连接密码
kubectl -n postgres get secret postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do -o 'jsonpath={.data.password}' | base64 -d- 1
5、部署 Minio 对象存储集群
hellm 方式部署minio 对象存储集群，至少需要4个节点，并创建一个名为 harbor 的 bucket
helm repo add minio https://helm.min.io/ helm upgrade --install minio minio/minio --namespace minio --create-namespace --set replicas=4 \ --set rootUser=minio \ --set rootPassword=minio123 \ --set service.type=NodePort \ --set consoleService.type=NodePort \ --set resources.requests.memory=1Gi \ --set buckets[0].name=harbor \ --set buckets[0].policy=upload \ --set buckets[0].purge=false- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
查看minio集群状态
root@node01:~# kubectl -n minio get pods NAME READY STATUS RESTARTS AGE minio-0 1/1 Running 0 5d6h minio-1 1/1 Running 0 5d3h minio-2 1/1 Running 0 5d6h minio-3 1/1 Running 0 5d6h- 1
- 2
- 3
- 4
- 5
- 6
获取minio service连接信息
root@node01:~# kubectl -n minio get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minio NodePort 10.96.3.193 <none> 9000:32000/TCP 5d6h minio-console NodePort 10.96.3.4 <none> 9001:32001/TCP 5d6h minio-svc ClusterIP None <none> 9000/TCP 5d6h- 1
- 2
- 3
- 4
- 5
6、部署 harbor 镜像仓库
添加helm repo
helm repo add harbor https://helm.goharbor.io- 1
自定义values.yaml配置参数，其中
192.168.72.50为节点IP地址。$ cat my-harbor-values.yaml expose: type: nodePort tls: auto: commonName: "192.168.72.50" externalURL: https://192.168.72.50:30003 nginx: replicas: 2 portal: replicas: 2 core: replicas: 2 jobservice: replicas: 2 registry: replicas: 2 chartmuseum: replicas: 2 trivy: replicas: 2 notary: server: replicas: 2 signer: replicas: 2 persistence: imageChartStorage: disableredirect: true type: s3 s3: region: us-west-1 bucket: harbor accesskey: minio secretkey: minio123 regionendpoint: http://minio.minio:9000 secure: false v4auth: true jobservice: jobLoggers: - database database: type: external external: host: "acid-minimal-cluster.postgres" port: "5432" username: "harbor" password: "H9AZVgIoXWUPgoYpQJq0Z3NoVNzxKPAZjZCApg3sUafl9lI0ixFtNGKlkeP2ieY8" coreDatabase: "registry" notaryServerDatabase: "notary_server" notarySignerDatabase: "notary_signer" sslmode: "require" redis: type: external external: addr: "rfs-redisfailover.redis:26379" sentinelMasterSet: "mymaster" coreDatabaseIndex: "0" jobserviceDatabaseIndex: "1" registryDatabaseIndex: "2" chartmuseumDatabaseIndex: "3" trivyAdapterIndex: "5" password: ""- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
部署harbor仓库
helm upgrade -i harbor harbor/harbor -n harbor --create-namespace -f my-harbor-values.yaml- 1
查看harbor状态
root@node01:~# kubectl -n harbor get pods NAME READY STATUS RESTARTS AGE harbor-chartmuseum-5d97bc8b8c-4ccl8 1/1 Running 0 8m54s harbor-chartmuseum-5d97bc8b8c-rw9n9 1/1 Running 0 8m54s harbor-core-6df5b674dc-2lgx4 1/1 Running 0 8m54s harbor-core-6df5b674dc-ws892 1/1 Running 0 8m54s harbor-jobservice-9b6d5fb5c-s96tp 1/1 Running 0 8m54s harbor-nginx-6d9cf558c5-gpxfw 1/1 Running 0 8m54s harbor-nginx-6d9cf558c5-wlq86 1/1 Running 0 8m54s harbor-notary-server-75cdb75f7d-44fkj 1/1 Running 0 8m54s harbor-notary-server-75cdb75f7d-bfh2w 1/1 Running 0 8m54s harbor-notary-signer-594587fb66-8c7sq 1/1 Running 0 8m54s harbor-notary-signer-594587fb66-zrl97 1/1 Running 0 8m54s harbor-portal-685498cc69-fc9js 1/1 Running 0 8m54s harbor-portal-685498cc69-nhzqj 1/1 Running 0 8m54s harbor-registry-6cc5d9d5b6-cxj6v 2/2 Running 0 8m54s harbor-registry-6cc5d9d5b6-dzzx2 2/2 Running 0 8m54s harbor-trivy-0 1/1 Running 0 8m54s harbor-trivy-1 1/1 Running 0 8m42s- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
查看service
root@node01:~# kubectl -n harbor get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE harbor NodePort 10.96.2.38 <none> 80:30002/TCP,443:30003/TCP,4443:30004/TCP 10m harbor-chartmuseum ClusterIP 10.96.1.174 <none> 80/TCP 10m harbor-core ClusterIP 10.96.2.160 <none> 80/TCP 10m harbor-jobservice ClusterIP 10.96.3.245 <none> 80/TCP 10m harbor-notary-server ClusterIP 10.96.3.157 <none> 4443/TCP 10m harbor-notary-signer ClusterIP 10.96.1.47 <none> 7899/TCP 10m harbor-portal ClusterIP 10.96.1.203 <none> 80/TCP 10m harbor-registry ClusterIP 10.96.3.197 <none> 5000/TCP,8080/TCP 10m harbor-trivy ClusterIP 10.96.3.229 <none> 8080/TCP 10m- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
登录harbor仓库，本地上传镜像测试
登录minio对象存储console控制台查看存储在bucket中的镜像
-
相关阅读:
8年软件测试工程师感悟——写给还在迷茫中的朋友
微信小程序开发前准备
Make.com实现多个APP应用的自动化的入门指南
ADS村田电感.mod（spice netlist文件）和.s2p模型导入与区别
第十一章《Java实战常用类》第6节：Object类
05 字符串
BAT 常用命令
.mkp勒索病毒解密方法|勒索病毒解决|勒索病毒恢复|数据库修复
Reflex WMS中阶系列7：已经完成拣货尚未Load的HD如果要取消拣货，该如何处理？
纪念陈皓（左耳朵耗子）
- 原文地址：https://blog.csdn.net/networken/article/details/126321152
