20.Redis系列之高可用集群模式

本文学习redis中集群模式，通过Redis Cluster部署拓扑进行水平扩展

1. Redis Cluster相关配置

# 设置redis实例能够加入集群
cluster-enabled yes
# 集群配置文件，由redis节点创建和更新
cluster-config-file nodes-6379.conf
# 节点超时时间15s
cluster-node-timeout 15000
# 集群内部通信的端口，即集群总线端口
cluster-port 16379
# 为了获得最大可用性，设置为0，这意味着副本将始终尝试故障转移
# 否则(node-timeout * cluster-replica-validity-factor) + repl-ping-replica-period时间后才会故障转移
cluster-replica-validity-factor 0
# 主节点至少拥有1个副本则宕机后副本才会故障迁移
cluster-migration-barrier 1
# 允许自动故障迁移
cluster-allow-replica-migration yes
# 即使当部分插槽不可用集群仍可用
cluster-require-full-coverage no
# 副本可以自动故障迁移
cluster-replica-no-failover no
# 集群挂掉不可读
cluster-allow-reads-when-down no

# 以下为docker部署特别指定
cluster-announce-ip 192.168.1.5
cluster-announce-port 6379
cluster-announce-bus-port 16379
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

以上配置我们假定为redis6379.conf、对于redis6380.conf、redis6381.conf、redis6382.conf、redis6383.conf、redis6384.conf，我们修改为对应端口即可，如下所示

cluster-announce-ip 192.168.1.5
cluster-announce-port 6380
cluster-announce-bus-port 16380
1
2
3

关注公众号 算法小生，回复 redis集群 即可获取对应代码详细配置

2. docker-compose配置及集群启动

version: '3'
services:
  redis6379:
    container_name: redis6379
    image: redis:7.0.5-alpine3.16
    ports:
      - 6379:6379
      - 16379:16379
    command: redis-server /usr/local/redis/conf/redis.conf
    volumes:
      - ./conf/redis6379.conf:/usr/local/redis/conf/redis.conf
    sysctls:
      - net.core.somaxconn=1024
    privileged: true
    environment:
      - TZ=Asia/Shanghai
      - LANG=en_US.UTF-8
  redis6380:
    container_name: redis6380
    image: redis:7.0.5-alpine3.16
    ports:
      - 6380:6379
      - 16380:16379
    command: redis-server /usr/local/redis/conf/redis.conf
    volumes:
      - ./conf/redis6380.conf:/usr/local/redis/conf/redis.conf
    sysctls:
      - net.core.somaxconn=1024
    privileged: true
    environment:
      - TZ=Asia/Shanghai
      - LANG=en_US.UTF-8
  redis6381:
    container_name: redis6381
    image: redis:7.0.5-alpine3.16
    ports:
      - 6381:6379
      - 16381:16379
    command: redis-server /usr/local/redis/conf/redis.conf
    volumes:
      - ./conf/redis6381.conf:/usr/local/redis/conf/redis.conf
    sysctls:
      - net.core.somaxconn=1024
    privileged: true
    environment:
      - TZ=Asia/Shanghai
      - LANG=en_US.UTF-8
  redis6382:
    container_name: redis6382
    image: redis:7.0.5-alpine3.16
    ports:
      - 6382:6379
      - 16382:16379
    command: redis-server /usr/local/redis/conf/redis.conf
    volumes:
      - ./conf/redis6382.conf:/usr/local/redis/conf/redis.conf
    sysctls:
      - net.core.somaxconn=1024
    privileged: true
    environment:
      - TZ=Asia/Shanghai
      - LANG=en_US.UTF-8
  redis6383:
    container_name: redis6383
    image: redis:7.0.5-alpine3.16
    ports:
      - 6383:6379
      - 16383:16379
    command: redis-server /usr/local/redis/conf/redis.conf
    volumes:
      - ./conf/redis6383.conf:/usr/local/redis/conf/redis.conf
    sysctls:
      - net.core.somaxconn=1024
    privileged: true
    environment:
      - TZ=Asia/Shanghai
      - LANG=en_US.UTF-8
  redis6384:
    container_name: redis6384
    image: redis:7.0.5-alpine3.16
    ports:
      - 6384:6379
      - 16384:16379
    command: redis-server /usr/local/redis/conf/redis.conf
    volumes:
      - ./conf/redis6384.conf:/usr/local/redis/conf/redis.conf
    sysctls:
      - net.core.somaxconn=1024
    privileged: true
    environment:
      - TZ=Asia/Shanghai
      - LANG=en_US.UTF-8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

执行docker-compose up -d启动redis集群

shenjian\redis\cluster>docker-compose up -d
WARNING: Found orphan containers (es3, kibana, es1) for this project. If you removed or renamed this service in your compose file, you can run this command with th
e --remove-orphans flag to clean it up.
Creating redis6382 ... done
Creating redis6379 ... done
Creating redis6384 ... done
Creating redis6380 ... done
Creating redis6383 ... done
Creating redis6381 ... done
1
2
3
4
5
6
7
8
9

我们连接6379节点，执行以下redis-cli --askpass -p 6379 cluster nodes命令查看节点集群信息

/data # redis-cli --askpass -p 6379 cluster nodes
Please input password: ******
e8392606e948e6e33183119a93b7b3011f5b9407 192.168.1.5:6379@16379 myself,master - 0 0 0 connected
1
2
3

执行以下命令进行集群创建

/data #  redis-cli --cluster create --cluster-replicas 1 192.168.1.5:6379 192.168.1.5:6380 192.168.1.5:6381 192.168.1.5:
6382 192.168.1.5:6383 192.168.1.5:6384 --askpass
Please input password: ******
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.5:6383 to 192.168.1.5:6379
Adding replica 192.168.1.5:6384 to 192.168.1.5:6380
Adding replica 192.168.1.5:6382 to 192.168.1.5:6381
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: e8392606e948e6e33183119a93b7b3011f5b9407 192.168.1.5:6379
   slots:[0-5460] (5461 slots) master
M: 3e55473bdff571c26761998a499da50dd743082b 192.168.1.5:6380
   slots:[5461-10922] (5462 slots) master
M: 76d7a639bc4849c60fbb0986389c8cf0a335e064 192.168.1.5:6381
   slots:[10923-16383] (5461 slots) master
S: 896b2f4d563abb071758506544b0db41365ffb3e 192.168.1.5:6382
   replicates e8392606e948e6e33183119a93b7b3011f5b9407
S: e89bbe523c5854a056356400960034c1715c11ee 192.168.1.5:6383
   replicates 3e55473bdff571c26761998a499da50dd743082b
S: 42a6b0e0f2df8a95fafa3d52d01a41e1b9310581 192.168.1.5:6384
   replicates 76d7a639bc4849c60fbb0986389c8cf0a335e064
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 192.168.1.5:6379)
M: e8392606e948e6e33183119a93b7b3011f5b9407 192.168.1.5:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: e89bbe523c5854a056356400960034c1715c11ee 192.168.1.5:6383
   slots: (0 slots) slave
   replicates 3e55473bdff571c26761998a499da50dd743082b
S: 42a6b0e0f2df8a95fafa3d52d01a41e1b9310581 192.168.1.5:6384
   slots: (0 slots) slave
   replicates 76d7a639bc4849c60fbb0986389c8cf0a335e064
S: 896b2f4d563abb071758506544b0db41365ffb3e 192.168.1.5:6382
   slots: (0 slots) slave
   replicates e8392606e948e6e33183119a93b7b3011f5b9407
M: 76d7a639bc4849c60fbb0986389c8cf0a335e064 192.168.1.5:6381
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 3e55473bdff571c26761998a499da50dd743082b 192.168.1.5:6380
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

再次执行集群查看命令

/data # redis-cli --askpass -p 6379 cluster nodes
Please input password: ******
e8392606e948e6e33183119a93b7b3011f5b9407 192.168.1.5:6379@16379 myself,master - 0 1669455837000 1 connected 0-5460
e89bbe523c5854a056356400960034c1715c11ee 192.168.1.5:6383@16383 slave 3e55473bdff571c26761998a499da50dd743082b 0 1669455833031 2 connected
42a6b0e0f2df8a95fafa3d52d01a41e1b9310581 192.168.1.5:6384@16384 slave 76d7a639bc4849c60fbb0986389c8cf0a335e064 0 1669455838063 3 connected
896b2f4d563abb071758506544b0db41365ffb3e 192.168.1.5:6382@16382 slave e8392606e948e6e33183119a93b7b3011f5b9407 0 1669455837056 1 connected
76d7a639bc4849c60fbb0986389c8cf0a335e064 192.168.1.5:6381@16381 master - 0 1669455835000 3 connected 10923-16383
3e55473bdff571c26761998a499da50dd743082b 192.168.1.5:6380@16380 master - 0 1669455836051 2 connected 5461-10922
1
2
3
4
5
6
7
8

我们可以看到3主3从集群已经创建成功，0-16383总共16384个插槽已经分配完毕，并且可以看到

• 0-5460插槽分配到主节点192.168.1.5:6379@16379
• 5461-10922插槽分配到主节点192.168.1.5:6380@16380
• 10923-16383插槽分配到主节点 192.168.1.5:6381@16381

3. 数据写入插槽

# 连接到集群
/data # redis-cli -c -p 6379 --askpass
Please input password: ******
# 设置键值，可以看到被分配到14315插槽位于192.168.1.5:6381
127.0.0.1:6379> set username shenjian
-> Redirected to slot [14315] located at 192.168.1.5:6381
OK
192.168.1.5:6381> set age 26
-> Redirected to slot [741] located at 192.168.1.5:6379
OK
# 多值设置key不在同一个插槽，报错，原因CRC16（key）% 16384得出插槽位置不唯一
192.168.1.5:6380> mset username shenjian age 20
(error) CROSSSLOT Keys in request don't hash to the same slot
# 我们通过哈希标签{}标识指定CRC16（key）% 16384中的key为user,故可设置成功
192.168.1.5:6380> mset username:{user} shenjian age:{user} 20
OK
# 获取值
192.168.1.5:6380> mget username:{user} age:{user}
1) "shenjian"
2) "20"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

4. 自动故障转移

我们把6379shutdown,连接其他节点查看集群状态可以看到6379主节点的丛节点192.168.1.5:6382@16382成为了主节点

/data # redis-cli --askpass -p 6379 cluster nodes
Please input password: ******
e89bbe523c5854a056356400960034c1715c11ee 192.168.1.5:6383@16383 slave 3e55473bdff571c26761998a499da50dd743082b 0 1669457824960 2 connected
3e55473bdff571c26761998a499da50dd743082b 192.168.1.5:6380@16380 myself,master - 0 1669457820000 2 connected 5461-10922
76d7a639bc4849c60fbb0986389c8cf0a335e064 192.168.1.5:6381@16381 master - 0 1669457823954 3 connected 10923-16383
42a6b0e0f2df8a95fafa3d52d01a41e1b9310581 192.168.1.5:6384@16384 slave 76d7a639bc4849c60fbb0986389c8cf0a335e064 0 1669457823000 3 connected
e8392606e948e6e33183119a93b7b3011f5b9407 192.168.1.5:6379@16379 master,fail - 1669457796163 1669457795760 1 connected
896b2f4d563abb071758506544b0db41365ffb3e 192.168.1.5:6382@16382 master - 0 1669457822948 7 connected 0-5460
1
2
3
4
5
6
7
8

由于我们是单机模拟伪集群，没有进行数据目录挂载，在实际的集群中，请勿忘记数据目录挂载备份