docker swarm 布署minio集群

活动地址：毕业季·进击的技术er

docker swarm 布署minio集群

环境：ubuntu18.04服务器 4台，docker(docker-compose，docker swarm)

分为主节点（1个）和从节点(3个)：

192.168.44.134 root1

192.168.44.135 root2

192.168.44.136 root3

192.168.44.137 root4

一、初始化主节点

1.主节点下执行以下命令

docker swarm init --advertise-addr 192.168.44.134
1

如果出现：Error response from daemon: --live-restore daemon configuration is incompatible with swarm mode

可尝试此解决方法：

vim /etc/docker/daemon.json
---
"live-restore": true
修改true为false
"live-restore": f
---
# 重启Docker
systemctl restart docker.service
1
2
3
4
5
6
7
8

后面若出现这个错也是一样

2.查看节点信息

docker node ls
1

二、将子节点添加到主节点swarm

1.在子节点运行docker swarm join ...

在docker swarm init中得到的输出docker swarm join --token SWMTKN-1-25i33klsk74p0oarwrbsyge4ggmrvto33o42iinm8oho1geksu-6vqqb5s56mdjrlpausbt2hhah 10.0.0.151:2377在各个子节点运行

若没有则在主节点执行docker swarm join-token worker,也可获得

2.在主节点查看节点信息

docker node ls
# MANAGER　显示为Leader是主节点，其余的为工作节点
1
2

三、在主节点为minio生成密钥

1.输入命令(也是设置账号和密码，后面yaml文件里面使用)

echo "admin" | docker secret create access_key -
echo "admin123456" | docker secret create secret_key -
# 分布式Minio里所有的节点需要有同样的access秘钥和secret秘钥，这样这些节点才能建立联接。为了实现这个，你需要在执行minio server命令之前，先将access秘钥和secret秘钥export成环境变量。
1
2
3

四、给节点打标签，主节点执行

# 节点打标签，绑定容器与节点之间一一对应的关系
docker node update --label-add minio1=true 7czsx0pl2u50bpjllg37dmvu1
docker node update --label-add minio2=true zx76va5pcadgblj8hnxms1x0z
docker node update --label-add minio3=true 64odnhwvebsjhw1tox721kyvx
docker node update --label-add minio4=true dz7idun0rtjcqx416y0jbwxvp
1
2
3
4
5

五、部署分布式minio服务

# 运行命令，部署minio，服务名：minio_stack,docker-compose-secrets.yaml文件见最后
docker stack deploy --compose-file=docker-compose-secrets.yaml minio_stack
1
2

部署成功后通过浏览器访问：http://IP:9001 来访问minio server客户端

查看正在运行的服务列表：

docker service ls
1

查看哪些节点正在运行该服务:

docker stack ps minio_stack
1

删除集群

docker stack rm minio_stack
1

六、在各个节点中都会运行一个容器

查看其中一个子节点：

docker ps
1

查看日志可以查看minio客户端地址（这里这个地址是容器内地址，可以映射出来）

docker logs --tail 100 -f <CONTAINED>
1

七、minio数据迁移

docker启动mc实例,并进入容器内部

docker run -it --entrypoint=/bin/sh minio/mc
1

通过mc命令连接两个minio服务

mc alias set minio1 http://127.0.0.1:9000 adminminio adminminio
mc alias set 名称 服务地址 用户名 密码
1
2

迁移数据

#全量迁移,重名文件不覆盖,bucket不存在会自动创建
mc mirror minio1 minio2
#只是迁移某个bucket,以test为例,目标的bucket需要提前建好
mc mirror minio1/test minio2/test
#覆盖重名文件,加--overwrite
mc mirror --overwrite minio1/test minio2/test
1
2
3
4
5
6

docker-compose-secrets.yaml

version: '3.3'

services:
  minio1:
    image: minio/minio:RELEASE.2022-04-16T04-26-02Z
    hostname: minio1
    volumes:
      - /home/liu/minio/data:/export
    ports:
      - "9000:9000"
      - "9001:9001"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio1==true
    command: server --address ':9000' --console-address ':9001' http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio2:
    image: minio/minio:RELEASE.2022-04-16T04-26-02Z
    hostname: minio2
    volumes:
      - /home/liu/minio/data:/export
    ports:
      - "9002:9000"
      - "9003:9001"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio2==true
    command: server --address ':9000' --console-address ':9001' http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio3:
    image: minio/minio:RELEASE.2022-04-16T04-26-02Z
    hostname: minio3
    volumes:
      - /home/liu/minio/data:/export
    ports:
      - "9004:9000"
      - "9005:9001"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio3==true
    command: server --address ':9000' --console-address ':9001' http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio4:
    image: minio/minio:RELEASE.2022-04-16T04-26-02Z
    hostname: minio4
    volumes:
      - /home/liu/minio/data:/export
    ports:
      - "9006:9000"
      - "9007:9001"
    networks:
      - minio_distributed
    deploy:
      restart_policy:
        delay: 10s
        max_attempts: 10
        window: 60s
      placement:
        constraints:
          - node.labels.minio4==true
    command: server --address ':9000' --console-address ':9001' http://minio{1...4}/export
    secrets:
      - secret_key
      - access_key
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
      
networks:
  minio_distributed:
    driver: overlay

secrets:
  secret_key:
    external: true
  access_key:
    external: true
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

参考链接:

https://www.cnblogs.com/lenovo_tiger_love/p/13050511.html

https://blog.csdn.net/lihongbao80/article/details/125070577