k8s部署三个节点zookeeper，1主2从集群

zookeeper（zk）在微服务应用中通常作为注册中心使用，内部自行选举leader，配置部署非常方便。

zk在k8s中部署需要注意事项：
1）zk集群是内部自行选举leader，少数服从多数原则，所以至少部署3个实列；
2）zk需要持久化数据存储，所以需要配置PV；
3）zk部署为有状态负载方式；
4）为了达到集群负载效果，zk实列不能部署到同一台主机（节点）上，所以使用强制反亲合（podAntiAffinity），保证各个pod在不同主机上进行部署。

zk部署脚本
1、创建PV
vi zk-pv.yaml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: k8s-pv-zk1
  annotations:
    volume.beta.kubernetes.io/storage-class: "anything"
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/lib/zookeeper"
  persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: k8s-pv-zk2
  annotations:
    volume.beta.kubernetes.io/storage-class: "anything"
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/lib/zookeeper"
  persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: k8s-pv-zk3
  annotations:
    volume.beta.kubernetes.io/storage-class: "anything"
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/var/lib/zookeeper"
  persistentVolumeReclaimPolicy: Recycle
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

执行部署PV脚本
kubectl apply -f zk-pv.yaml

2、zk实列部署
vi zookeeper.yaml

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  namespace: default
  labels:
    app: zk
spec:
  selector:
    app: zk
  clusterIP: None
  ports:
  - name: server
    port: 2888
  - name: leader-election
    port: 3888
--- 
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  namespace: default
  labels:
    app: zk
spec:
  selector:
    app: zk
  type: NodePort
  ports:
  - name: client
    port: 2181
    nodePort: 31811
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
  namespace: default
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
  namespace: default
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: "zk-hs"
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: zk # has to match .spec.selector.matchLabels
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - {key: app, operator: In, values: ["zk"]}
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: zk
        imagePullPolicy: Always
        image: chaotingge/zookeeper:kubernetes-zookeeper1.0-3.4.10
        resources:
          requests:
            memory: "500Mi"
            cpu: "0.5"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
        --servers=3 \
        --data_dir=/var/lib/zookeeper/data \
        --data_log_dir=/var/lib/zookeeper/data/log \
        --conf_dir=/opt/zookeeper/conf \
        --client_port=2181 \
        --election_port=3888 \
        --server_port=2888 \
        --tick_time=2000 \
        --init_limit=10 \
        --sync_limit=5 \
        --heap=512M \
        --max_client_cnxns=60 \
        --snap_retain_count=3 \
        --purge_interval=12 \
        --max_session_timeout=40000 \
        --min_session_timeout=4000 \
        --log_level=INFO"
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/zookeeper
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.beta.kubernetes.io/storage-class: "anything"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

执行脚本
kubectl apply -f zookeeper.yaml

核心脚本解释：

apiVersion: apps/v1
kind: StatefulSet  # 将 Deployment调整为有状态的StatefulSet部署方式
metadata:
  name: zk
  namespace: default
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: "zk-hs"
  replicas: 3 # 集群部署至少3个实列
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:                # 配置亲和性
        podAntiAffinity:       # 强制使用反亲和 保证实列部署在不同主机上
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - {key: app, operator: In, values: ["zk"]} # 匹配规则 只要app=zk的都不能部署在一起
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: zk
        imagePullPolicy: Always
        image: chaotingge/zookeeper:kubernetes-zookeeper1.0-3.4.10
        resources:
          requests:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

验证集群是否成功：

# 验证集群是否成功，登录pod容器
kubectl exec -it <pod实列名称> /bin/sh
# 验证当前zk实列是主节点（leader）还是从节点（follower）
zkServer.sh status
# 登录zkCli客户端
zkCli.sh
# 创建一个节点 查看各个节点是否能同步到数据
create /test-zk Yang douya
# 在其他节点执行查看命令，验证是否通过/test-zk获取到数据Yang douya
get /test-zk
1
2
3
4
5
6
7
8
9
10