k8s搭建 rabbitmq集群

rabbitmq镜像下载-CSDN
mq镜像下载位置
链接：https://pan.baidu.com/s/1tJ1xxfA174Kjj4CYcjFJ8w?pwd=cyr4
提取码：cyr4

1. 创建挂载使用的pv以及pvc

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    cattle.io/creator: norman
  name: rabbit-cluster-pv
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Ti
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: rabbit-cluster-pvc
    namespace: cloud-test
    resourceVersion: "304986615"
  nfs:
    path: /home/nfs/rabbit-cluster
    server: 192.168.1.26
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    cattle.io/creator: norman
  name: rabbit-cluster-pvc
  namespace: cloud-test
  resourceVersion: "304986620"
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Ti
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: rabbit-cluster-pvc
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Ti
  phase: Bound

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

2. 创建对应的集群服务和无头服务

kind: Service
apiVersion: v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-headless
  namespace: cloud-test
spec:
  clusterIP: None
  ports:
  - name: rmqport
    port: 5672
    targetPort: 5672
  selector:
    app: rabbitmq-cluster

---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster
  namespace: cloud-test
spec:
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: 15672
  - name: rmqport
    port: 5672
    targetPort: 5672
  selector:
    app: rabbitmq-cluster
  type: ClusterIP
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

3. 创建rabbitmq需要的RBAC权限

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq
  namespace: cloud-test
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq
  namespace: cloud-test
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq
  namespace: cloud-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rabbitmq
subjects:
- kind: ServiceAccount
  name: rabbitmq
  namespace: cloud-test
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

4. 创建rabbitmq的configmap配置文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-cluster-config
  namespace: cloud-test
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  enabled_plugins: |
    [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_delayed_message_exchange].
  rabbitmq.conf: |-
    default_user = mqadmin  ##账户
    default_pass = mqadmin  ##密码
    cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s  ##开启rabbit集群
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.node_cleanup.interval = 30
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    queue_master_locator=min-masters
    loopback_users.guest = false
    ## cluster_formation.randomized_startup_delay_range.min = 0
    ## cluster_formation.randomized_startup_delay_range.max = 2
    # 必须设置service_name，否则Pod无法正常启动，这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量
    #cluster_formation.k8s.service_name = rabbitmq-headless
    # default is rabbitmq-cluster's namespace
    # hostname_suffix  必须设置hostname_suffix，否则节点不能成为集群,注意修改rabbitmq为ns名称
    cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.cloud-test.svc.cluster.local
    # 内存上限
    vm_memory_high_watermark.absolute = 512MB
    # 硬盘上限
    disk_free_limit.absolute = 100GB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

5. 创建rabbitmq集群

apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  generation: 1
  labels:
    cattle.io/creator: norman
    app: rabbitmq-cluster
  name: rabbitmq-cluster
  namespace: cloud-test
spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: rabbitmq-cluster
  serviceName: rabbitmq-headless
  template:
    metadata:
      labels:
        app: rabbitmq-cluster
    spec:
      containers:
      - env:
        - name: K8S_SERVICE_NAME    ##集群内部的无头服务
          value: rabbitmq-headless
        - name: MY_POD_NAME     ##容器名称
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE    ##命名空间
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: K8S_HOSTNAME_SUFFIX      ##集群加入路径
          value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_ERLANG_COOKIE
          value: mycookie
        - name: RABBITMQ_NODENAME           ##节点名称
          value: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        image: 192.168.0.14/project/kubernetes-rabbitmq-autocluster:3.7.8
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          failureThreshold: 3
          initialDelaySeconds: 160
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 15
        name: rabbitmq-cluster
        ports:
        - containerPort: 15672
          name: 15672tcp00
          protocol: TCP
        - containerPort: 5672
          name: 5672tcp00
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          failureThreshold: 3
          initialDelaySeconds: 20
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 10
        resources: {}
        securityContext:
          capabilities: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/rabbitmq
          name: config-volume
        - mountPath: /var/lib/rabbitmq/mnesia
          name: rabbitmq-data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: rabbitmq
      serviceAccountName: rabbitmq
      terminationGracePeriodSeconds: 10
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf     ##mq配置
          - key: enabled_plugins
            path: enabled_plugins    ##插件
          name: rabbitmq-cluster-config   ##configmap
        name: config-volume
      - name: rabbitmq-data
        persistentVolumeClaim:
          claimName: rabbit-cluster-pvc  ##pvc
  updateStrategy:
    type: OnDelete
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

6. 检查rabbitmq集群

注意！！！
如果这个过程会有点长，耐心等待几分钟，如果长时间没有加入集群，请删除后重新部署 rabbitmq的有状态服务
6.1 通过查看日志确认有没有加入集群

kubectl logs rabbitmq-cluster-0 -n cloud-test
1

I1112 15:00:22.372930   19510 http.go:113] HTTP2 has been explicitly disabled
2022-11-12 06:15:00.345 [info] <0.33.0> Application lager started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.447 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.593 [info] <0.33.0> Application inets started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.593 [info] <0.33.0> Application jsx started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.593 [info] <0.33.0> Application recon started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.615 [info] <0.33.0> Application os_mon started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.615 [info] <0.33.0> Application xmerl started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.615 [info] <0.33.0> Application crypto started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.616 [info] <0.33.0> Application cowlib started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.616 [info] <0.33.0> Application asn1 started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.616 [info] <0.33.0> Application public_key started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.760 [info] <0.33.0> Application ssl started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.780 [info] <0.33.0> Application ranch started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.790 [info] <0.33.0> Application cowboy started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.790 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.790 [info] <0.33.0> Application rabbit_common started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:01.791 [info] <0.198.0> 
 Starting RabbitMQ 3.7.8 on Erlang 20.3.8.5
 Copyright (C) 2007-2018 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/

  ##  ##
  ##  ##      RabbitMQ 3.7.8. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: 

              Starting broker...
2022-11-12 06:15:01.828 [info] <0.198.0> 
 node           : rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : XhdCf8zpVJeJ0EHyaxszPg==
 log(s)         : >
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local
2022-11-12 06:15:06.551 [info] <0.206.0> Memory high watermark set to 488 MiB (512000000 bytes) of 15885 MiB (16657137664 bytes) total
2022-11-12 06:15:06.565 [info] <0.208.0> Enabling free disk space monitoring
2022-11-12 06:15:06.565 [info] <0.208.0> Disk free limit set to 100000MB
2022-11-12 06:15:06.575 [info] <0.211.0> Limiting to approx 1048476 file handles (943626 sockets)
2022-11-12 06:15:06.576 [info] <0.212.0> FHC read buffering:  OFF
2022-11-12 06:15:06.576 [info] <0.212.0> FHC write buffering: ON
2022-11-12 06:15:06.578 [info] <0.198.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch...
2022-11-12 06:15:06.578 [info] <0.198.0> Configured peer discovery backend: rabbit_peer_discovery_k8s
2022-11-12 06:15:06.578 [info] <0.198.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s
2022-11-12 06:15:06.578 [info] <0.198.0> Peer discovery backend does not support locking, falling back to randomized delay
2022-11-12 06:15:06.578 [info] <0.198.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay.
2022-11-12 06:15:06.604 [info] <0.198.0> k8s endpoint listing returned nodes not yet ready: rabbitmq-cluster-0
2022-11-12 06:15:06.604 [info] <0.198.0> All discovered existing cluster peers: 
2022-11-12 06:15:06.604 [info] <0.198.0> Discovered no peer nodes to cluster with
2022-11-12 06:15:06.607 [info] <0.33.0> Application mnesia exited with reason: stopped
2022-11-12 06:15:06.643 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:06.812 [info] <0.198.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-11-12 06:15:06.852 [info] <0.198.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-11-12 06:15:06.895 [info] <0.198.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2022-11-12 06:15:06.895 [info] <0.198.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping registration.
2022-11-12 06:15:06.911 [info] <0.198.0> Priority queues enabled, real BQ is rabbit_variable_queue
2022-11-12 06:15:06.916 [info] <0.395.0> Starting rabbit_node_monitor
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: 1 to apply
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: No durable queues found. Skipping message store migration
2022-11-12 06:15:06.969 [info] <0.198.0> message_store upgrades: Removing the old message store data
2022-11-12 06:15:06.994 [info] <0.198.0> message_store upgrades: All upgrades applied successfully
2022-11-12 06:15:07.039 [info] <0.198.0> Management plugin: using rates mode 'basic'
2022-11-12 06:15:07.041 [info] <0.198.0> Adding vhost '/'
2022-11-12 06:15:07.048 [info] <0.437.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2022-11-12 06:15:07.052 [info] <0.437.0> Starting message stores for vhost '/'
2022-11-12 06:15:07.053 [info] <0.441.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2022-11-12 06:15:07.059 [info] <0.437.0> Started message store of type transient for vhost '/'
2022-11-12 06:15:07.059 [info] <0.444.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2022-11-12 06:15:07.061 [warning] <0.444.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2022-11-12 06:15:07.065 [info] <0.437.0> Started message store of type persistent for vhost '/'
2022-11-12 06:15:07.067 [info] <0.198.0> Creating user 'mqadmin'
2022-11-12 06:15:07.068 [info] <0.198.0> Setting user tags for user 'mqadmin' to [administrator]
2022-11-12 06:15:07.069 [info] <0.198.0> Setting permissions for 'mqadmin' in '/' to '.*', '.*', '.*'
2022-11-12 06:15:07.072 [info] <0.482.0> started TCP Listener on [::]:5672
2022-11-12 06:15:07.082 [info] <0.198.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.092 [info] <0.198.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.092 [info] <0.33.0> Application rabbit started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.496.0> Peer discovery: enabling node cleanup (will only log warnings). Check interval: 60 seconds.
2022-11-12 06:15:07.093 [info] <0.33.0> Application rabbitmq_peer_discovery_common started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.33.0> Application rabbitmq_web_dispatch started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.33.0> Application rabbitmq_peer_discovery_k8s started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.093 [info] <0.33.0> Application amqp_client started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.096 [info] <0.33.0> Application rabbitmq_management_agent started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.137 [info] <0.549.0> Management plugin started. Port: 15672
2022-11-12 06:15:07.137 [info] <0.655.0> Statistics database started.
2022-11-12 06:15:07.139 [info] <0.33.0> Application rabbitmq_management started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
2022-11-12 06:15:07.139 [info] <0.33.0> Application rabbitmq_delayed_message_exchange started on node 'rabbit@rabbitmq-cluster-0.rabbitmq-headless.cloud-test.svc.cluster.local'
 completed with 6 plugins.
2022-11-12 06:15:07.428 [info] <0.5.0> Server startup complete; 6 plugins started.
 * rabbitmq_delayed_message_exchange
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_web_dispatch
 * rabbitmq_peer_discovery_common
2022-11-12 06:16:20.847 [info] <0.395.0> node 'rabbit@rabbitmq-cluster-1.rabbitmq-headless.cloud-test.svc.cluster.local' up
2022-11-12 06:16:21.393 [info] <0.395.0> rabbit on node 'rabbit@rabbitmq-cluster-1.rabbitmq-headless.cloud-test.svc.cluster.local' up
2022-11-12 06:17:41.304 [info] <0.395.0> node 'rabbit@rabbitmq-cluster-2.rabbitmq-headless.cloud-test.svc.cluster.local' up
2022-11-12 06:17:41.851 [info] <0.395.0> rabbit on node 'rabbit@rabbitmq-cluster-2.rabbitmq-headless.cloud-test.svc.cluster.local' up

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102

6.2 通过进入容器确认有没有加入集群

kubectl exec -it rabbitmq-cluster-0 bash -n cloud-test
1

7. 最终对外暴露出rabbitmq的两个端口，访问rabbit的dashboard页面

8. 开启rabbitmq的镜像集群模式