Redis5.x搭建集群

仅在同一台服务器测试

1、安装redis

安装redis完成之后复制六份

# 创建集群存放redis的文件夹
mkdir redis-cluster

cd ../

cp -r redis/ redis-cluster/redis-9301
cp -r redis/ redis-cluster/redis-9302
cp -r redis/ redis-cluster/redis-9303
cp -r redis/ redis-cluster/redis-9304
cp -r redis/ redis-cluster/redis-9305
cp -r redis/ redis-cluster/redis-9306

cd redis-cluster
1
2
3
4
5
6
7
8
9
10
11
12
13

2、编写redis.conf配置文件

# 编写公共redis.conf文件
vim redis.conf

# 注释该行使其他电脑可以访问
# bind 127.0.0.1  
 
# 改为yes，使实例后台运行
daemonize yes 
 
# 开启集群支持
cluster-enabled yes 
 
# 集群节点不可用的最长时间，超过该时间进行从库故障转移
cluster-node-timeout 15000

# 指定为外网IP
cluster-announce-ip 59.110.xx.xxx

# 设置密码
requirepass 123456
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

进入复制的每个redis文件夹中修改master 的 redis.conf文件为：

# 拼接公共redis.conf文件
include /usr/local/redis/redis-cluster/redis.conf
pidfile "/var/run/redis_9301.pid"
port 9301
dbfilename "dump9301.rdb"

# 指定节点文件名称
cluster-config-file nodes-9301.conf
1
2
3
4
5
6
7
8

再修改slave redis中修改redis.conf文件为：

# 拼接公共redis.conf文件
include /usr/local/redis/redis-cluster/redis.conf
pidfile "/var/run/redis_9304.pid"
port 9304
dbfilename "dump9304.rdb"

# 指定节点文件名称
cluster-config-file nodes-9304.conf

# 设置密码
masterauth 123456
1
2
3
4
5
6
7
8
9
10
11

9301、9302、9303为master

9304、9305、9306为slave

3、编写启动脚本、创建集群

启动脚本

#!/bin/bash
redis_path="/usr/local/redis/redis-cluster/"
cd ${redis_path}
ps aux | grep redis | grep cluster | grep -v grep | awk '{print $2}' | xargs kill -9
# ps -ef | grep -v grep | grep redis |  awk -F' ' '{print $2}' | xargs kill -9
cluster_num=`ps aux | grep redis | grep cluster | wc -l`
if [ "${cluster_num}" -le 0 ]
then
        echo -e "===== Success: Has killed all cluster progress."
else
        echo -e "===== Fail: There still are ${cluster_num} is alive.\n"
        exit 1
fi

rm -rf ${redis_path}redis*/dump*
rm -rf ${redis_path}redis*/nodes*
data_num=`find ${redis_path} -type f | grep -E "dump*|nodes*" | wc -l`
if [ "${data_num}" -le 0 ]
then
        echo -e "===== Success: Has remove all dump.rdb and nodes configure file."
else
        echo -e "===== Fail: There still are files is exist,Use command: \n\tfind ${redis_path} -type f | grep -E \"dump.rbd|nodes*\" \nto search, and then remove them manually.\n"
        exit 1
fi

cd redis-9301/
./redis-server redis.conf
cd ../redis-9302/
./redis-server redis.conf
cd ../redis-9303/
./redis-server redis.conf
cd ../redis-9304/
./redis-server redis.conf
cd ../redis-9305/
./redis-server redis.conf
cd ../redis-9306/
./redis-server redis.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

创建集群脚本

#!/bin/bash
# 创建集群
cd /usr/local/redis/redis-5.0.14/src
./redis-cli --cluster create --cluster-replicas 1 59.110.xx.xxx:9301 59.110.xx.xxx:9302 59.110.xx.xxx:9303 59.110.xx.xxx:9304 59.110.xx.xxx:9305 59.110.xx.xxx:9306 -a 123456
1
2
3
4

–cluster create: 创建集群

–cluster-relicas: 集群副本数。 这里是1，是1主机1从机的模式，如果设置为2（即：2台从机）会失败。因为集群中至少要有3个主机，所以设置2台从机时，至少需要9个节点才可以。

最后的参数中列出全部的redis主机IP地址和端口号。

-a: 后边跟的是redis的密码

4、启动

# 启动各redis节点
./restart_redis.sh
# 创建集群
[root@hcg redis-cluster]# ./create_cluster.sh 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 59.110.xx.xxx:9305 to 59.110.xx.xxx:9301
Adding replica 59.110.xx.xxx:9306 to 59.110.xx.xxx:9302
Adding replica 59.110.xx.xxx:9304 to 59.110.xx.xxx:9303
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: b78b59b8cadb3e1f77d520a164597777e2bc18f8 59.110.xx.xxx:9301
   slots:[0-5460] (5461 slots) master
M: dc1cdaa729ebe03c7b9702025426c2b86df70da2 59.110.xx.xxx:9302
   slots:[5461-10922] (5462 slots) master
M: 651576fcd8fb66e83212351331bfb110b3a3a524 59.110.xx.xxx:9303
   slots:[10923-16383] (5461 slots) master
S: e7cf094e16c520f2d1faf9ae65f68e97c1fba562 59.110.xx.xxx:9304
   replicates dc1cdaa729ebe03c7b9702025426c2b86df70da2
S: 8a61f7e96b7160127921c9f0a9e4e9f65963d71f 59.110.xx.xxx:9305
   replicates 651576fcd8fb66e83212351331bfb110b3a3a524
S: 52eed0573d4b9f20f0b61434bebfec05ce8dcac5 59.110.xx.xxx:9306
   replicates b78b59b8cadb3e1f77d520a164597777e2bc18f8
Can I set the above configuration? (type 'yes' to accept): 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

输入yes

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 59.110.xx.xxx:9301)
M: b78b59b8cadb3e1f77d520a164597777e2bc18f8 59.110.xx.xxx:9301
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 52eed0573d4b9f20f0b61434bebfec05ce8dcac5 59.110.xx.xxx:9306
   slots: (0 slots) slave
   replicates b78b59b8cadb3e1f77d520a164597777e2bc18f8
S: 8a61f7e96b7160127921c9f0a9e4e9f65963d71f 59.110.xx.xxx:9305
   slots: (0 slots) slave
   replicates 651576fcd8fb66e83212351331bfb110b3a3a524
M: 651576fcd8fb66e83212351331bfb110b3a3a524 59.110.xx.xxx:9303
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: e7cf094e16c520f2d1faf9ae65f68e97c1fba562 59.110.xx.xxx:9304
   slots: (0 slots) slave
   replicates dc1cdaa729ebe03c7b9702025426c2b86df70da2
M: dc1cdaa729ebe03c7b9702025426c2b86df70da2 59.110.xx.xxx:9302
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@hcg redis-cluster]# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

5、访问客户端

./redis-cli -p 9301 -a 123456 -c
59.110.xx.xx:9302> 
1
2

1、-p：集群中任意一个redis服务的端口号

2、-a：redis的密码

3、-c：集群模式

可使用：cluster nodes 命令查看集群状态，查询结果如下：

59.110.xx.xxx:9302> cluster nodes 
b78b59b8cadb3e1f77d520a164597777e2bc18f8 59.110.xx.xxx:9301@19301 master - 0 1659782936979 1 connected 0-5460
e7cf094e16c520f2d1faf9ae65f68e97c1fba562 59.110.xx.xxx:9304@19304 slave dc1cdaa729ebe03c7b9702025426c2b86df70da2 0 1659782935977 4 connected
8a61f7e96b7160127921c9f0a9e4e9f65963d71f 59.110.xx.xxx:9305@19305 slave 651576fcd8fb66e83212351331bfb110b3a3a524 0 1659782934976 5 connected
dc1cdaa729ebe03c7b9702025426c2b86df70da2 59.110.xx.xxx:9302@19302 myself,master - 0 1659782935000 2 connected 5461-10922
651576fcd8fb66e83212351331bfb110b3a3a524 59.110.xx.xxx:9303@19303 master - 0 1659782934000 3 connected 10923-16383
52eed0573d4b9f20f0b61434bebfec05ce8dcac5 59.110.xx.xxx:9306@19306 slave b78b59b8cadb3e1f77d520a164597777e2bc18f8 0 1659782936000 6 connected
59.110.xx.xxx:9302> 
1
2
3
4
5
6
7
8

6、整合spring boot

1、application.yml配置文件：

spring:
  # redis集群
  redis:
    password: 123456
    cluster:
      nodes:
        - 59.110.xx.xxx:9301
        - 59.110.xx.xxx:9302
        - 59.110.xx.xxx:9303
        - 59.110.xx.xxx:9304
        - 59.110.xx.xxx:9305
        - 59.110.xx.xxx:9306
      max-redirects: 3
    # 连接超时时间
    timeout: 60s
    lettuce:
      pool:
        # 连接池中的最小空闲连接
        min-idle: 0
        # 连接池中的最大空闲连接
        max-idle: 8
        # 连接池的最大数据库连接数
        max-active: 8
        # #连接池最大阻塞等待时间（使用负值表示没有限制）
        max-wait: -1m
      cluster:
        refresh:
          period: 60s
          adaptive: true
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

2、配置类：

@Configuration
@EnableCaching
public class RedisConfig extends CachingConfigurerSupport
{
    @Bean
    @SuppressWarnings(value = { "unchecked", "rawtypes" })
    public RedisTemplate<Object, Object> redisTemplate(RedisConnectionFactory connectionFactory)
    {
        RedisTemplate<Object, Object> template = new RedisTemplate<Object, Object>();
        template.setConnectionFactory(connectionFactory);

        FastJson2JsonRedisSerializer serializer = new FastJson2JsonRedisSerializer(Object.class);

        ObjectMapper mapper = new ObjectMapper();
        mapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
        mapper.activateDefaultTyping(LaissezFaireSubTypeValidator.instance, ObjectMapper.DefaultTyping.NON_FINAL, JsonTypeInfo.As.PROPERTY);
        serializer.setObjectMapper(mapper);

        // 使用StringRedisSerializer来序列化和反序列化redis的key值
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(serializer);

        // Hash的key也采用StringRedisSerializer的序列化方式
        template.setHashKeySerializer(new StringRedisSerializer());
        template.setHashValueSerializer(serializer);

        template.afterPropertiesSet();
        return template;
    }

    @Bean
    public DefaultRedisScript<Long> limitScript()
    {
        DefaultRedisScript<Long> redisScript = new DefaultRedisScript<Long>();
        redisScript.setScriptText(limitScriptText());
        redisScript.setResultType(Long.class);
        return redisScript;
    }

    @Bean
    public LettuceClientConfigurationBuilderCustomizer clientConfigurationBuilderCustomizer(){
        return clientConfigurationBuilder -> clientConfigurationBuilder.readFrom(ReadFrom.REPLICA_PREFERRED);
    }

    /**
     * 限流脚本
     * @return
     */
    private String limitScriptText()
    {
        return "local key = KEYS[1]\n" +
                "local count = tonumber(ARGV[1])\n" +
                "local time = tonumber(ARGV[2])\n" +
                "local current = redis.call('get', key);\n" +
                "if current and tonumber(current) > count then\n" +
                "    return tonumber(current);\n" +
                "end\n" +
                "current = redis.call('incr', key)\n" +
                "if tonumber(current) == 1 then\n" +
                "    redis.call('expire', key, time)\n" +
                "end\n" +
                "return tonumber(current);";
    }

    @Bean
    public DefaultClientResources lettuceClientResources() {
        return DefaultClientResources.create();
    }

    @Bean
    @SuppressWarnings("all")
    public LettuceConnectionFactory lettuceConnectionFactory(RedisProperties redisProperties, ClientResources clientResources) {

        ClusterTopologyRefreshOptions topologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
                .enablePeriodicRefresh(Duration.ofSeconds(30)) //按照周期刷新拓扑
                .enableAllAdaptiveRefreshTriggers() //根据事件刷新拓扑
                .build();

        ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
                //redis命令超时时间,超时后才会使用新的拓扑信息重新建立连接
                .timeoutOptions(TimeoutOptions.enabled(Duration.ofSeconds(10)))
                .topologyRefreshOptions(topologyRefreshOptions)
                .build();

        LettuceClientConfiguration clientConfiguration = LettuceClientConfiguration.builder()
                .clientResources(clientResources)
                .clientOptions(clusterClientOptions)
                .build();

        RedisClusterConfiguration clusterConfig = new RedisClusterConfiguration(redisProperties.getCluster().getNodes());
        clusterConfig.setMaxRedirects(redisProperties.getCluster().getMaxRedirects());
        clusterConfig.setPassword(RedisPassword.of(redisProperties.getPassword()));

        return new LettuceConnectionFactory(clusterConfig, clientConfiguration);
    }

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

7、安装过程中的问题：

springboot2.3.0后支持通过配置文件和配置LettuceConnectionFactory中设置自动刷新和周期性刷新，采用三主三从的cluster模式

Q: springboot 启动连接报错，Unable to connect to 127.0.0.1；但是项目配置文件写的是外网ip，是正确的
A：客户端lettuce在初始化集群会loadProperties，读到的远端redis集群信息是127.0.0.1的；
解决方法：创建集群，指定ip

./redis-cli --cluster create --cluster-replicas 1 59.110.xx.xxx:9301 59.110.xx.xxx:9302 59.110.xx.xxx:9303 59.110.xx.xxx:9304 59.110.xx.xxx:9305 59.110.xx.xxx:9306 -a 123456
1

Q：指定为外网ip创建遇到一个新问题，创建集群，使用外网ip无法创建，Waiting for the cluster to join…
A：暂时先检查一下安全组的出入方向、服务器防火墙，搜索发现，还需要开放集群总线的端口，“Redis端口+10000”，即19301-19306。

Q：本机使用java客户端连接，报Unable to connect to：172.xx.xx.xx ，发现连接的内网ip。
A：redis的配置文件应该指定cluster-announce-ip 为外网ip：59.110.xx.xxx