redis cluster集群搭建

集群搭建

启动6个redis实例

创建6份配置文件

mkdir redis-cluster
cd redis-cluster
mkdir 7001 7002 7003 8001 8002 8003
1
2
3

7001文件夹创建配置文件redis.conf

port 7001
cluster-enabled yes
cluster-config-file nodes-7001.conf
cluster-node-timeout 5000
appendonly yes
1
2
3
4
5

然后copy到其他文件夹，记得改端口号，还有cluster-config-file的值，这个值不能重，必须唯一，否则无法启动

➜  redis git:(unstable) ✗ tree redis-cluster
redis-cluster
├── 7001
│   └── redis.conf
├── 7002
│   └── redis.conf
├── 7003
│   └── redis.conf
├── 8001
│   └── redis.conf
├── 8002
│   └── redis.conf
└── 8003
    └── redis.conf

7 directories, 6 files 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

启动6个独立的redis实例

nohup ./src/redis-server ./redis-cluster/7001/redis.conf &
nohup ./src/redis-server ./redis-cluster/7002/redis.conf &
nohup ./src/redis-server ./redis-cluster/7003/redis.conf &
nohup ./src/redis-server ./redis-cluster/8001/redis.conf &
nohup ./src/redis-server ./redis-cluster/8002/redis.conf &
nohup ./src/redis-server ./redis-cluster/8003/redis.conf &
1
2
3
4
5
6

确认是否全部启动成功

➜  redis git:(unstable) ✗  ps -ef |grep redis
  502 54910 84839   0  4:04下午 ttys007    0:01.25 ./redis-server *:7001 [cluster]
  502 63857 84839   0  4:17下午 ttys007    0:00.13 ./src/redis-server *:7002 [cluster]
  502 64177 84839   0  4:18下午 ttys007    0:00.08 ./src/redis-server *:7003 [cluster]
  502 64551 84839   0  4:18下午 ttys007    0:00.06 ./src/redis-server *:8001 [cluster]
  502 64651 84839   0  4:18下午 ttys007    0:00.04 ./src/redis-server *:8002 [cluster]
  502 64721 84839   0  4:18下午 ttys007    0:00.04 ./src/redis-server *:8003 [cluster]
1
2
3
4
5
6
7

启动redis cluster

–cluster-replicas 1表示每个master一个slave

➜  redis git:(unstable) ✗ ./src/redis-cli --cluster create 127.0.0.1:7001 127.0.0.1:7002 \
127.0.0.1:7003 127.0.0.1:8001 127.0.0.1:8002 127.0.0.1:8003 \
--cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:8002 to 127.0.0.1:7001
Adding replica 127.0.0.1:8003 to 127.0.0.1:7002
Adding replica 127.0.0.1:8001 to 127.0.0.1:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: da719074003da0628451b51ea31b3dc33531df9d 127.0.0.1:7001
   slots:[0-5460] (5461 slots) master
M: 031ebc9d3abd215ae467d1e9ec801a1664e2aeca 127.0.0.1:7002
   slots:[5461-10922] (5462 slots) master
M: e92d35f68da0727360113754dca6000de8fb19bb 127.0.0.1:7003
   slots:[10923-16383] (5461 slots) master
S: a1b535c7207cfcf3556917a663020d1ad588e086 127.0.0.1:8001
   replicates e92d35f68da0727360113754dca6000de8fb19bb
S: fc3b476515919bc2d26651a3589098f6f7b2862c 127.0.0.1:8002
   replicates da719074003da0628451b51ea31b3dc33531df9d
S: b69892d37edd69069a0ba39a97adc3818e4d45a6 127.0.0.1:8003
   replicates 031ebc9d3abd215ae467d1e9ec801a1664e2aeca
Can I set the above configuration? (type 'yes' to accept): yes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

通过这段日志，可以发现，redis cluster 把16384个slot预分配给了3对主从节点。

查看集群节点信息

127.0.0.1:7001> cluster nodes
d0a93b9a335e3a5be1d8a94fc99bf353d9ac9666 127.0.0.1:7001@17001 myself,master - 0 1697100496000 1 connected 0-5460
e7f17d098f930cd6095ffdafb182a0bda214485f 127.0.0.1:8003@18003 slave d0a93b9a335e3a5be1d8a94fc99bf353d9ac9666 0 1697100497553 1 connected
a7e5325784d8fea48491df0eef8e9b27f1b29797 127.0.0.1:7002@17002 master - 0 1697100497553 2 connected 5461-10922
c0ea6d485168f69eda3aaa7956a67f252a7db8a1 127.0.0.1:8002@18002 slave 3a3211ae21ae8f03ba9ea4b1a7f2d0064582461b 0 1697100496844 3 connected
34f5d00d5b3f5ca5a75228ca93a2049eaf34a6df 127.0.0.1:8001@18001 slave a7e5325784d8fea48491df0eef8e9b27f1b29797 0 1697100497857 2 connected
3a3211ae21ae8f03ba9ea4b1a7f2d0064582461b 127.0.0.1:7003@17003 master - 0 1697100496541 3 connected 10923-16383
1
2
3
4
5
6
7

测试集群的读写

 src git:(unstable) ✗ ./redis-cli -c -p 7001
127.0.0.1:7001> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7003
OK
127.0.0.1:7003> get foo
"bar"
127.0.0.1:7003> set hello world
-> Redirected to slot [866] located at 127.0.0.1:7001
OK
127.0.0.1:7001> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7003
"bar"
127.0.0.1:7003> get hello
-> Redirected to slot [866] located at 127.0.0.1:7001
"world"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

测试failover功能

先停掉一个master 7001

src git:(unstable) ✗ ps -ef |grep redis
  502 77572 34733   0  4:38下午 ttys005    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox redis
  502 54910 84839   0  4:04下午 ttys007    0:03.01 ./redis-server *:7001 [cluster]
  502 63857 84839   0  4:17下午 ttys007    0:01.92 ./src/redis-server *:7002 [cluster]
  502 64177 84839   0  4:18下午 ttys007    0:01.86 ./src/redis-server *:7003 [cluster]
  502 64551 84839   0  4:18下午 ttys007    0:01.75 ./src/redis-server *:8001 [cluster]
  502 64651 84839   0  4:18下午 ttys007    0:01.83 ./src/redis-server *:8002 [cluster]
  502 64721 84839   0  4:18下午 ttys007    0:01.86 ./src/redis-server *:8003 [cluster]
➜  src git:(unstable) ✗ kill -9 54910
1
2
3
4
5
6
7
8
9

观察节点信息

➜  src git:(unstable) ✗ ./redis-cli -c -p 7002
127.0.0.1:7002> set lvsheng  a
-> Redirected to slot [694] located at 127.0.0.1:8002
127.0.0.1:8002> cluster nodes
031ebc9d3abd215ae467d1e9ec801a1664e2aeca 127.0.0.1:7002@17002 master - 0 1694077364000 2 connected 5461-10922
a1b535c7207cfcf3556917a663020d1ad588e086 127.0.0.1:8001@18001 slave e92d35f68da0727360113754dca6000de8fb19bb 0 1694077364649 3 connected
b69892d37edd69069a0ba39a97adc3818e4d45a6 127.0.0.1:8003@18003 slave 031ebc9d3abd215ae467d1e9ec801a1664e2aeca 0 1694077364548 2 connected
fc3b476515919bc2d26651a3589098f6f7b2862c 127.0.0.1:8002@18002 myself,master - 0 1694077364000 7 connected 0-5460
da719074003da0628451b51ea31b3dc33531df9d 127.0.0.1:7001@17001 master,fail - 1694075900491 1694075898000 1 disconnected
e92d35f68da0727360113754dca6000de8fb19bb 127.0.0.1:7003@17003 master - 0 1694077364000 3 connected 10923-16383
127.0.0.1:8002> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:7
cluster_stats_messages_ping_sent:4100
cluster_stats_messages_pong_sent:4129
cluster_stats_messages_meet_sent:1
cluster_stats_messages_fail_sent:5
cluster_stats_messages_auth-req_sent:5
cluster_stats_messages_sent:8240
cluster_stats_messages_ping_received:4125
cluster_stats_messages_pong_received:4100
cluster_stats_messages_auth-ack_received:2
cluster_stats_messages_received:8227
total_cluster_links_buffer_limit_exceeded:0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

依然是3个主节点，但是7001已经失联，8002从slave变成了master。