doris operator部署Doris集群教程

前言

kubernetes Operator是遵循kubernetes API和控制器模式，它主要用来封装运维业务逻辑的软件。它利用kubernetes的自定义资源定义（CRD）扩展API，并通过控制器模式监听资源对象，实现应用的部署和生命周期。大多数人使用kubernetes的时候是使用原生资源（pod、deployment、Service等），当然也可以使用operator来特定需求的新业务逻辑。
kubernetes Operator组成主要包括以下几个部分：
1.自定义资源定义（CRD）：CRD 是对kubernetes API的扩展，允许用户定义自己的资源类型和属性。Operator使用CRD来定义自己管理的应用资源对象。
2.控制器（Controller）：控制器监视自定义资源对象的状态，并根据定义的业务逻辑对资源对象进行协调，让状态达到预期。
3.管理器（manager）:它本质上就是一个管理器框架。主要负责启动所有控制器，使多个控制器可以协同工作。而且它还提供共享缓存和客户端，不同控制器可以共享这些资源。实现了控制器逻辑的并发和同步。
Doris 官网上有 Kubernetes 部署的文档，无奈根据官网的文档，构建完镜像无法成功启动。故参考官网做了一些改动，成功启动 FE、BE 节点。此探索为临时部署方案，还需再完善、优化。目前好消息就是，Doris提供的Operator可以方便部署Doris集群。

部署流程

部署非常简单，首先在github下载Doris的operator项目，链接在doris operator。然后，根据operator部署Doris cluster，在本篇文章中，例子有三个节点,具体如下所示。

doris cluster的yaml文件如下

# this yaml is the simplest mode to deploy doris with fe and be.
# the log and be storage use `EmptyDir` mode as disk. when the pod restart the log and storage will lost.
# This yaml only for experience deploy and usability on k8s about doris.

apiVersion: doris.selectdb.com/v1
kind: DorisCluster
metadata:
  labels:
    app.kubernetes.io/name: doriscluster
    app.kubernetes.io/instance: doriscluster-sample
    app.kubernetes.io/part-of: doris-operator
  name: doriscluster
  namespace: doris
spec:
  feSpec:
    replicas: 1
    image: selectdb/doris.fe-ubuntu:2.0.2
    envVars:
      - name: TZ
        value: "Asia/Shanghai"
    configMapInfo:
      # use kubectl create configmap fe-configmap --from-file=fe.conf
      configMapName: fe-configmap
      resolveKey: fe.conf
    service:
      type: NodePort
      servicePorts: 
      - nodePort: 31030
        targetPort: 9030
      - nodePort: 31031
        targetPort: 8030
    requests:
      memory: 1Gi
    limits:
      memory: 2Gi
    persistentVolumes:
    - mountPath: /opt/apache-doris/fe/doris-meta
      name: fe-data
      persistentVolumeClaimSpec:
        # when use specific storageclass, the storageClassName should reConfig, example as annotation.
        storageClassName: doris-fe-storage
        accessModes:
        - ReadWriteOnce
        resources:
          # notice: if the storage size less 5G, fe will not start normal.
          requests:
            storage: 5Gi
  beSpec:
    replicas: 2
    image: selectdb/doris.be-ubuntu:2.0.2
    envVars:
      - name: TZ
        value: "Asia/Shanghai"
    systemInitialization:
      command: ["/sbin/sysctl", "-w", "vm.max_map_count=2000000"]
    configMapInfo:
      # use kubectl create configmap be-configmap --from-file=be.conf
      configMapName: be-configmap
      resolveKey: be.conf
    persistentVolumes:
    - mountPath: /opt/apache-doris/be/storage
      name: be-data
      persistentVolumeClaimSpec:
        # when use specific storageclass, the storageClassName should reConfig, example as annotation.
        storageClassName: doris-be-storage
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 6Gi

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

operator的yaml文件定义如下

# permissions to do leader election.
apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: doris-operator
    app.kubernetes.io/name: namespace
    app.kubernetes.io/instance: doris
    app.kubernetes.io/component: doris-operator
    app.kubernetes.io/part-of: doris-operator
  name: doris
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/name: role
    app.kubernetes.io/instance: leader-election-role
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: doris-operator
    app.kubernetes.io/part-of: doris-operator
    app.kubernetes.io/managed-by: kustomize
  name: leader-election-role
  namespace: doris
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - patch
      - delete
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - patch
      - delete
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/name: rolebinding
    app.kubernetes.io/instance: leader-election-rolebinding
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: doris-operator
    app.kubernetes.io/part-of: doris-operator
    app.kubernetes.io/managed-by: kustomize
  name: leader-election-rolebinding
  namespace: doris
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: leader-election-role
subjects:
  - kind: ServiceAccount
    name: doris-operator
    namespace: doris
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: doris-operator
rules:
  - apiGroups:
      - apps
    resources:
      - statefulsets
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - apps
    resources:
      - statefulsets/status
    verbs:
      - get
  - apiGroups:
      - autoscaling
    resources:
      - horizontalpodautoscalers
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - serviceaccounts
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - doris.selectdb.com
    resources:
      - dorisclusters
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - doris.selectdb.com
    resources:
      - dorisclusters/finalizers
    verbs:
      - update
  - apiGroups:
      - doris.selectdb.com
    resources:
      - dorisclusters/status
    verbs:
      - get
      - patch
      - update
  - apiGroups:
      - rbac.authorization.k8s.io
    resources:
      - clusterrolebindings
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
  - apiGroups:
      - rbac.authorization.k8s.io
    resources:
      - rolebindings
    verbs:
      - create
      - delete
      - get
      - list
      - patch
      - update
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/name: clusterrolebinding
    app.kubernetes.io/instance: doris-operator-rolebinding
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: doris-operator
    app.kubernetes.io/part-of: doris-operator
    app.kubernetes.io/managed-by: kustomize
  name: doris-operator-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: doris-operator
subjects:
  - kind: ServiceAccount
    name: doris-operator
    namespace: doris
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: serviceaccount
    app.kubernetes.io/instance: controller-doris-operator-sa
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: doris-operator
    app.kubernetes.io/part-of: doris-operator
    app.kubernetes.io/managed-by: kustomize
  name: doris-operator
  namespace: doris
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: doris-operator
  namespace: doris
  labels:
    control-plane: doris-operator
    app.kubernetes.io/name: deployment
    app.kubernetes.io/instance: doris-operator
    app.kubernetes.io/component: doris-operator
    app.kubernetes.io/created-by: doris-operator
    app.kubernetes.io/part-of: doris-operator
spec:
  selector:
    matchLabels:
      control-plane: doris-operator
  replicas: 1
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: doris-operator
      labels:
        control-plane: doris-operator
    spec:
      # TODO(user): Uncomment the following code to configure the nodeAffinity expression
      # according to the platforms which are supported by your solution.
      # It is considered best practice to support multiple architectures. You can
      # build your manager image using the makefile target docker-buildx.
      # affinity:
      #   nodeAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       nodeSelectorTerms:
      #         - matchExpressions:
      #           - key: kubernetes.io/arch
      #             operator: In
      #             values:
      #               - amd64
      #               - arm64
      #               - ppc64le
      #               - s390x
      #           - key: kubernetes.io/os
      #             operator: In
      #             values:
      #               - linux
      securityContext:
        runAsNonRoot: true
        # TODO(user): For common cases that do not require escalating privileges
        # it is recommended to ensure that all your Pods/Containers are restrictive.
        # More info: https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
        # Please uncomment the following code if your project does NOT have to work on old Kubernetes
        # versions < 1.19 or on vendors versions which do NOT support this field by default (i.e. Openshift < 4.11 ).
        # seccompProfile:
        #   type: RuntimeDefault
      containers:
        - command:
            - /dorisoperator
          args:
            - --leader-elect
          image: selectdb/doris.k8s-operator:latest
          name: dorisoperator
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - "ALL"
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8081
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8081
            initialDelaySeconds: 5
            periodSeconds: 10
          # TODO(user): Configure the resources accordingly based on the project requirements.
          # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
          resources:
            limits:
              cpu: "2"
              memory: 4Gi
            requests:
              cpu: "1"
              memory: 2Gi
      serviceAccountName: doris-operator
      terminationGracePeriodSeconds: 10
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342

fe的storageclass的yaml文件如下

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: doris-fe-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
1
2
3
4
5
6
7

fe的pv定义yaml文件如下

apiVersion: v1
kind: PersistentVolume
metadata:
   name: fe-node-71
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: doris-fe-storage
   local:
      path: /data/doris/fe
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - 172.16.43.71

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

be的storageclass定义yaml文件如下

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: doris-be-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
1
2
3
4
5
6
7

be的pv定义yaml文件如下

apiVersion: v1
kind: PersistentVolume
metadata:
   name: be-node-72
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: doris-be-storage
   local:
      path: /data/doris/be/storage
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - 172.16.43.72
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: be-node-73
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: doris-be-storage
   local:
      path: /data/doris/be/storage
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - 172.16.43.73
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

执行如下构建命令构建

kubectl apply -f doris-fe-st.yaml
kubectl apply -f doris-fe-pv.yaml
kubectl apply -f doris-be-st.yaml
kubectl apply -f doris-be-pv.yaml
kubectl apply -f operator.yaml -ndoris
kubectl apply -f doriscluster.yaml -ndoris
1
2
3
4
5
6

查看pvc确定是否绑定成功

查看pod 是否启动成功

执行show frontends命令查看fe的状态，结果如下

执行show backends命令查看be的状态，具体如下图