基于helm的方式在k8s集群中部署gitlab - 备份恢复（二）

接上一篇 基于helm的方式在k8s集群中部署gitlab - 部署（一），本篇重点介绍在k8s集群中备份gitlab的数据，并在虚拟机上部署相同版本的gitlab，然后将备份的数据进行还原恢复

1. 备份

由于使用的是minio对象存储，然后gitlab的ingress 使用的是nodeport的方式，而默认的minio的配置是域名配置，由于gitlab在14.9以后使用的tootlbox来进行备份的，因此需要更改toolbox引入minio的configmap配置文件。

但是每次upgrade后会覆盖cm文件，因为后面依然需要修改cm，或者使用外置minio。

# 查看tootbox的cm
kubectl get cm -n jihulab
# 编辑tootbox的cm
kubectl edit cm -n jihulab gitlab-toolbox
1
2
3
4

 ...
 ...
    if [ ! -f "/${secret_dir}/objectstorage/.s3cfg" ]; then
    cat < "/${secret_dir}/.s3cfg"
    [default]
    access_key = $(cat /init-secrets/minio/accesskey)
    secret_key = $(cat /init-secrets/minio/secretkey)
    bucket_location = us-east-1
    host_base = minio.bdeet.top:31501  #修改为nodeport的端口
    host_bucket = minio.bdeet.top:31501/%(bucket)   #修改为nodeport的端口
    default_mime_type = binary/octet-stream
    enable_multipart = True
    multipart_max_chunks = 10000
    multipart_chunk_size_mb = 128
    recursive = True
    recv_chunk = 65536
    send_chunk = 65536
    server_side_encryption = False
    signature_v2 = True
    socket_timeout = 300
    use_mime_magic = False
    verbosity = WARNING
    website_endpoint = https://minio.bdeet.top:31501  #修改为nodeport的端口
    EOF
...
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

然后delete掉toolbox的pod，执行备份

# 删除pod
kubectl delete pod -n jihulab gitlab-toolbox-7b796575d8-gplhc
# 备份
kubectl exec -it gitlab-toolbox-7b796575d8-7q8mh -n jihulab -- backup-utility
1
2
3
4

minio上备份的gitlab数据

2. 恢复到虚拟机上的gitlab

此处跳过安装gitlab到虚拟机上的操作，默认已经安装相同版本的gitlab服务

2.1 将minio上的备份文件（gitlab-backups）下载下来

2.2 将文件放在相同版本gitlab实例的backup目录下，然后执行restore命令

参考gitlab恢复

cd /var/opt/gitlab/backups
sudo gitlab-backup restore
1
2

2.3 配置gitlab实例的对象存储

编辑gitlab.rb文件

...
...
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = true
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'region' => 'us-east-1',
  'path_style' => 'true',
  'host' => 'minio.bdeet.top:30476',
  'endpoint' => 'https://minio.bdeet.top:30476',
  'aws_access_key_id' => 'NHsiBL6v589G4h1JTn2Kj2sFAV5SxyVLslmoDSWdepqzRs6yYMic3QuKQvTPIXvW',
  'aws_secret_access_key' => 'ye3ySpmaaxCVADAhGz1MbhyBwWnGXW8iJEelVidvq1PZS1fYv6SoQjuTIvZHgHIj'
}

gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-mr-diffs'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
gitlab_rails['object_store']['objects']['backups']['bucket'] = 'gitlab-backups'
gitlab_rails['object_store']['objects']['backups']['tmpBucket'] = 'tmp'
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

gitlab-ctl reconfigure
1

2.4 gitlab 界面查看并测试

k8s上的项目

之前的文件可以看见，后面上传的文件也可以上传成功。

3. 最终形态的values文件

...
...
## 域名配置
  hosts:	
    domain: bdeet.top	
    hostSuffix:	
    externalIP:	
    ssh:	
    gitlab:	
      name: kube.bdeet.top	
      https: true	
    minio:	
      name: minio.bdeet.top	
      https: true	
    registry:	
      name: registry.bdeet.top	
      https: true
...
...
## ldap集成
	    ldap:	
      preventSignin: false	
      servers:	
        main:	
         label: 'LDAP'	
         host: '129.226.208.223'	
         port: 389	
         uid: 'uid'	
         bind_dn: 'cn=ldap,dc=wkx,dc=cn'	
         base: 'dc=wkx,dc=cn'	
         password:	
           secret: ldap-admin	
           key: password	
         encryption: 'plain'
...
...
## 配置邮箱       
	smtp:	
    enabled: true	
    address: smtp.gmail.com	
    port: 587	
    user_name: "kxw12108@gmail.com"	
    ## https://docs.gitlab.com/charts/installation/secrets#smtp-password	
    password:	
      secret: "smtp-gitlab"	
      key: password	
    # domain:	
    authentication: "login"	
    starttls_auto: true	
    openssl_verify_mode: "peer"	
    pool: false	
  ## https://docs.gitlab.com/charts/charts/globals#outgoing-email	
  ## Email persona used in email sent by GitLab	
  email:	
    from: "kxw12108@gmail.com"	
    display_name: "GitLab Administrator"	
    reply_to: "kxw12108@gmail.com"	
    subject_suffix: "GitLab"	
    smime:	
      enabled: false	
      secretName: ""	
      keyName: "tls.key"	
      certName: "tls.crt"
...
...
...
...
nginx-ingress:
  enabled: true
  ...
  ...
    service:
      externalTrafficPolicy: "Local"
      type: "NodePort" #ingress的svc修改为nodeport
    ...
    ...
...
...
  runner:
    registrationToken:
      secret: gitlab-gitlab-runner-secret  # gitlab-runner的secret
...
...
gitlab-runner:
  install: true
  gitlabUrl: https://kube.bdeet.top  #修改gitlab的域名
  rbac:
    create: true
  runners:
    privileged: true #开启特权
    locked: false
    config: |
      [[runners]]
        [runners.kubernetes]
        image = "ubuntu:18.04"
        {{- if .Values.global.minio.enabled }}
        [runners.cache]
          Type = "s3"
          Path = "gitlab-runner"
          Shared = true
          [runners.cache.s3]
            #ServerAddress = {{ include "gitlab-runner.cache-tpl.s3ServerAddress" . }}
            ServerAddress = "https://minio.bdeet.top:31501" #接入对象存储
            BucketName = "runner-cache"
            BucketLocation = "us-east-1"
            Insecure = false
...
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108