ansible部署ceph集群

一.  环境准备：

1.  环境分配：

2.  配置hosts解析，三个节点都要配置：

[root@node2 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.188.112 node2
192.168.188.113 node3
192.168.188.114 node4

3.  关闭防火墙和selinux：

[root@node2 ~]# selinuxsystemctl stop firewalld 
 
[root@node2 ~]# systemctl disable firewalld
 
[root@node2 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
 
[root@node2 ~]# setenforce 0 

4.  设置三个节点时间同步：

[root@node3 ~]# yum install chrony -y
 
[root@node3 ~]# cat /etc/chrony.conf 
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server ntp1.aliyun.com iburst
 
[root@node3 ~]# systemctl enable chronyd --now
 
[root@node3 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 120.25.115.20                 2  10   137   500  -1643us[-1954us] +/-   26ms

5.  配置三个节点的免密登录：

## 生成密钥文件
[root@node2 ~]# ssh-keygen -f ~/.ssh/id_rsa -P '' -q
 
[root@node2 ~]# ssh-copy-id node2
 
[root@node2 ~]# ssh-copy-id node3
 
[root@node2 ~]# ssh-copy-id node4

6.  配置扩展源：

[root@node2 ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com
/repo/epel-7.repo

7.  安装python2-pip，ansible，git：

[root@node2 ~]# yum install python2-pip ansible git -y

二.  部署ceph集群，在一个节点操作即可：

1.  克隆存储库：

[root@node2 ~]# git clone https://github.com/ceph/ceph-ansible.git
 
[root@node2 ~]# cd ceph-ansible
 
[root@node2 ceph-ansible]# git checkout stable-4.0

2.  安装ansible依赖包：

[root@node2 ceph-ansible]# yum install python-netaddr python-six -y

3.  修改hosts文件，添加主机清单：

[root@node2 ceph-ansible]# vim /etc/ansible/hosts
[mons]
node[2:4]
 
[osds]
node[2:4]
 
[mgrs]
node2
 
[mdss]
node[2:4]
 
[clients]
node[2:4]
 
[rgws]
node[2:4]
 
[grafana-server]
node2

4.  将" ceph-ansible/group_vars/ "下的文件全部备份，修改" ceph-ansible/group_vars/all.yml "配置：

[root@node2 ceph-ansible]# cd group_vars/
 
[root@node2 group_vars]# egrep -v '#|^$' all.yml
修改后的内容：
---
dummy:
mon_group_name: mons
osd_group_name: osds
rgw_group_name: rgws
mds_group_name: mdss
client_group_name: clients
mgr_group_name: mgrs
grafana_server_group_name: grafana-server
configure_firewall: False
ceph_origin: repository
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://mirrors.aliyun.com/ceph
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
ceph_stable_release: nautilus
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"
public_network: "192.168.188.0/24"
cluster_network: "192.168.188.0/24"
monitor_interface: eth0
osd_auto_discovery: true
osd_objectstore: filestore
radosgw_interface: eth0
dashboard_admin_password: ans123456
grafana_admin_password: admin
pg_autoscale_mode: True

5.  修改 " ceph-ansible/group_vars/osds.yml "配置：

[root@node2 group_vars]# vim osds.yml
.......
devices:
  - /dev/sdb
.......

6.  先备份" ceph-ansible/site.yml "，再修改" ceph-ansible/site.yml "配置：

[root@node2 ceph-ansible]# vim site.yml 
---
 
- hosts:
  - mons 
  - osds
  - mdss
  - rgws
  #- nfss
  #- rbdmirrors
  - clients
  - mgrs
  #- iscsigws
  - grafana-server
......

7.  运行" ceph-ansible/site.yml "文件即可：

[root@node2 ceph-ansible]# ansible-playbook -i /etc/ansible/hosts site.yml

8.  查看集群健康状态

## 查看集群健康状态时，可能会出现两个错误
[root@node2 cephy-ansible]# ceph health
HEALTH_WARN 1 pools have too few placement groups; mons are allowing insecure 
global_id reclaim 
 
1、HEALTH_WARN:pools have too many placement groups 
 
解决方法∶ceph mgr module disable pg_autoscaler
 
2、HEALTH_WARN: mons are allowing insecure global_id reclaim 
 
需要禁用掉不安全的模式，使用如下命令：
ceph config set mon auth_allow_insecure_global_id_reclaim false
 
[root@node2 cephy-ansible]# ceph health
HEALTH_OK
 
## 查看集群配置：
[root@node2 cephy-ansible]# ceph -s

9.  查看访问ceph的web页面的方式：

[root@node2 cephy-ansible]# ceph mgr services