基于docker搭建redis高可用集群

github学习笔记

Redis 集群搭建以及槽

1) Redis集群使用数据分片(sharding)而非一致性哈希(consistencyhashing)来实现:
2) 一个Redis集群包含16384个哈希槽(hashslot)，数据库中的每个键都属于这16384个哈希槽的其中一个，集群使用公式CRC16(key)%16384来计算键key于哪个槽，其中CRC16(key)语句用于计算键key的CRC16校验和
3) 将一个哈希槽从一个节点移动到另一个节点不会造成节点阻塞，所以无论是添加新节点还是移除已存在节点，又或者改变某个节点包含的哈希槽数量，都不会造成集群下线。
4) 对象保存到Redis之前先经过CRC16哈希到一个指定的Node上
5) 每个Node被平均分配了一个Slot段，对应着0-16383，Slot不能重复也不能缺失，否则会导致对象重复存储或无法存储。
6) Node之间也互相监听，一旦有Node退出或者加入，会按照Slot为单位做数据的迁移
    Node1如果掉线了，0-5640这些Slot将会平均分摊到Node2和Node3上

优点:
    将Redis的写操作分摊到了多个节点上，提高写的并发能力，扩容简单。
    
缺点:
    每个Node承担着互相监听、高并发数据写入、高并发数据读出，工作任务繁重

1
2
3
4
5
6
7
8
9
10
11
12
13
14

集群数据分片存储示意图

集群搭建(三主三从)*

# 修改配置文件
# 设置密码
masterauth $password
requirepass $password

# 开启集群
cluster-enabled yes

# 集群连接超时时间
cluster-node-timeout 15000

# 集群配置文件
cluster-config-file  nodes-6379.conf
1
2
3
4
5
6
7
8
9
10
11
12
13

修改完成启动容器

# master
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-master1 -v /root/redis-cluster/cluster-master1.conf:/usr/local/etc/redis/redis.conf -d -p 16300:6379 redis /bin/bash
9a9ce8cda97c962c6a909f3f8db6ac58211230893fb7d749de6d21ef7c7e3c9d
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-master2 -v /root/redis-cluster/cluster-master2.conf:/usr/local/etc/redis/redis.conf -d -p 16301:6379 redis /bin/bash
daf06c8ef1095ba2aa9690aaec815661b85de4c4eb76cfb84737b716ddc93d49
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-master2 -v /root/redis-cluster/cluster-master3.conf:/usr/local/etc/redis/redis.conf -d -p 16302:6379 redis /bin/bash

# slave
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-master3 -v /root/redis-cluster/cluster-master3.conf:/usr/local/etc/redis/redis.conf -d -p 16302:6379 redis /bin/bash
3acc354a13c089cadb6f814e4f710838fc61c7aae34e07ae433500ec80cb61b9
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-slave1 -v /root/redis-cluster/cluster-slave1.conf:/usr/local/etc/redis/redis.conf -d -p 26300:6379 redis /bin/bash
55a4d2c5eab3e7d950f6c3d27c28d9b78427e43b0bf279584dcdf73c3a02f858
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-slave2 -v /root/redis-cluster/cluster-slave2.conf:/usr/local/etc/redis/redis.conf -d -p 26301:6379 redis /bin/bash
e510eb5a12f7d0c60f63693fb5a672d813e1b522aa438f1c2179833aef792216
[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker run -it --name cluster-slave3 -v /root/redis-cluster/cluster-slave3.conf:/usr/local/etc/redis/redis.conf -d -p 26302:6379 redis /bin/bash
71b45258c4b1ea08f7f3d55d84c0c54c253fb13839abb6698f7611d2c4558f07


# 进入容器启动服务，同样的方式启动其他的容器，可以看到都是集群模式启动
[root@iZ2ze58f53sxjm9z7mgn5xZ ~]# docker exec -it cluster-master1 bash
root@9a9ce8cda97c:/data#  redis-server /usr/local/etc/redis/redis.conf
15:C 17 Sep 2022 07:16:47.827 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15:C 17 Sep 2022 07:16:47.827 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=15, just started
15:C 17 Sep 2022 07:16:47.827 # Configuration loaded
15:M 17 Sep 2022 07:16:47.827 * monotonic clock: POSIX clock_gettime
15:M 17 Sep 2022 07:16:47.828 * No cluster configuration found, I'm 13d15a762dd88c9389e54bac0d5a294d9b13f7af
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 7.0.4 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                  
 (    '      ,       .-`  | `,    )     Running in cluster mode            # 集群模式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

查看容器ip启动集群

[root@iZ2ze58f53sxjm9z7mgn5xZ redis-cluster]# docker inspect cluster-master1 | grep Address
            "LinkLocalIPv6Address": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "GlobalIPv6Address": "",
            "IPAddress": "172.17.0.2",
            "MacAddress": "02:42:ac:11:00:02",
                    "IPAddress": "172.17.0.2",
                    "GlobalIPv6Address": "",
                    "MacAddress": "02:42:ac:11:00:02",

# cluster-master1: "IPAddress": "172.17.0.2",
# cluster-master2: "IPAddress": "172.17.0.3",
# cluster-master3: "IPAddress": "172.17.0.4",

# cluster-slave1: "IPAddress": "172.17.0.8",
# cluster-slave2: "IPAddress": "172.17.0.9",
# cluster-slave3: "IPAddress": "172.17.0.10",
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

进入任意一个容器启动集群

# redis-cli --cluster create 172.17.0.2:6379  172.17.0.3:6379 172.17.0.4:6379 172.17.0.8:6379 172.17.0.9:6379 172.17.0.10:6379 --cluster-replicas 1 -a 123456

[root@iZ2ze58f53sxjm9z7mgn5xZ ~]# docker exec -it cluster-master1 bash
root@9a9ce8cda97c:/data# redis-cli --cluster create 172.17.0.2:6379  172.17.0.3:6379 172.17.0.4:6379 172.17.0.8:6379 172.17.0.9:6379 172.17.0.10:6379 --cluster-replicas 1 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
# 槽的分配信息
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
# 主从信息
Adding replica 172.17.0.9:6379 to 172.17.0.2:6379
Adding replica 172.17.0.10:6379 to 172.17.0.3:6379
Adding replica 172.17.0.8:6379 to 172.17.0.4:6379
M: 162d79c5a770d9e37fab29ccd0a04bc51f7755ea 172.17.0.2:6379
   slots:[0-5460] (5461 slots) master
M: 930f1af1f2876aca79e0d8f627e18fc20b16d120 172.17.0.3:6379
   slots:[5461-10922] (5462 slots) master
M: 63e637f635280682b0f6e24de2906126c0a7bda2 172.17.0.4:6379
   slots:[10923-16383] (5461 slots) master
S: 59f88c0e858e189524ac65a0e2b84d6dc7cece51 172.17.0.8:6379
   replicates 63e637f635280682b0f6e24de2906126c0a7bda2
S: 1820f990ccba0e1c929d497ef181ea83aeef7a8b 172.17.0.9:6379
   replicates 162d79c5a770d9e37fab29ccd0a04bc51f7755ea
S: 61d6982faa26d1e121d7be4e1fbb0c39f918cdf0 172.17.0.10:6379
   replicates 930f1af1f2876aca79e0d8f627e18fc20b16d120
   # 是否确认配置，必须输入yes，不能输入y
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.17.0.2:6379)
M: 162d79c5a770d9e37fab29ccd0a04bc51f7755ea 172.17.0.2:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 63e637f635280682b0f6e24de2906126c0a7bda2 172.17.0.4:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 930f1af1f2876aca79e0d8f627e18fc20b16d120 172.17.0.3:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 1820f990ccba0e1c929d497ef181ea83aeef7a8b 172.17.0.9:6379
   slots: (0 slots) slave
   replicates 162d79c5a770d9e37fab29ccd0a04bc51f7755ea
S: 59f88c0e858e189524ac65a0e2b84d6dc7cece51 172.17.0.8:6379
   slots: (0 slots) slave
   replicates 63e637f635280682b0f6e24de2906126c0a7bda2
S: 61d6982faa26d1e121d7be4e1fbb0c39f918cdf0 172.17.0.10:6379
   slots: (0 slots) slave
   replicates 930f1af1f2876aca79e0d8f627e18fc20b16d120
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

集群测试

# 进入任意master容器
[root@iZ2ze58f53sxjm9z7mgn5xZ ~]# docker exec -it cluster-master1 bash

root@9a9ce8cda97c:/data# redis-cli -h 172.17.0.2 -p 6379 -a 123456               # 正常启动命令
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.17.0.2:6379> keys *
(empty array)
172.17.0.2:6379> set uname python
(error) MOVED 10359 172.17.0.3:6379                 # 无法进行设置，显示当前的这个key的hash不属于当前这个结点

# 尝试登录它提示槽位的机器，就可以正常set
root@9a9ce8cda97c:/data# redis-cli -h 172.17.0.3 -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.17.0.3:6379> set uname python
OK


# 这样非常不方便，所以换一种方式启动,启动时加上参数-c
root@9a9ce8cda97c:/data# redis-cli -c -h 172.17.0.2 -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.17.0.2:6379> set uname java
# 自动进行分配到哪一个结点上，不需要手动切换结点
-> Redirected to slot [10359] located at 172.17.0.3:6379
OK
172.17.0.3:6379> set address beijing
-> Redirected to slot [3680] located at 172.17.0.2:6379
OK

# 查看写入的master对应的slave是否同步了数据
root@71b45258c4b1:/data# redis-cli -c -h 172.17.0.9 -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
172.17.0.9:6379> keys *
1) "address"
172.17.0.9:6379> get address
-> Redirected to slot [3680] located at 172.17.0.2:6379
"beijing"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

有任何问题敬请一起交流