• 【运维知识大神篇】两种方法，一键部署ElasticSearch集群（Shell+Ansible自动化部署）

    本篇文章给大家介绍一下用Shell和Ansible两种方式，去一键部署ES（ElasticSearch）集群，实现在集群部署中的批量化操作，享受自动化部署的魅力。

    准备主机

    主机名IPcpudiskmemory
    Ansible10.0.0.612C20G2G
    ELK10410.0.0.1042C20G2G
    ELK10510.0.0.1052C20G2G
    ELK10610.0.0.1062C20G2G

    Shell一键部署

    我们在Ansible上准备好所需文件（无JDK的ES安装包和oracle的JDK），在Ansible编写Shell脚本，实现ES集群的一键部署，其实脚本编写的思路有很多，这里我使用先做免密钥，做完后写for循环，里面进行单点部署，也可以直接部署出来一个节点后进行拷贝

    如果想修改目录的话需要修改脚本，elasticsearch.yml、es7.service里的文件，所以配置文件都打包放在了文末。

    1. [root@Ansible ~]# cat deploy_ES.sh
    2. #!/bin/bash
    3. #1、先做免密钥，方便文件的推送以及远程执行命令
    4. if [ -f /root/.ssh/id_rsa.pub ];then
    5. 	echo "公钥已经存在"
    6. else
    7. 	ssh-keygen -t rsa
    8. 	echo "公钥不存在，自动创建公钥"
    9. fi
    10.  
    11. hosts=(
    12. 10.0.0.104
    13. 10.0.0.105
    14. 10.0.0.106
    15. )
    16.  
    17. for i in ${hosts[*]}
    18. do
    19. 	expect <<EOF
    20.       set timeout 10
    21.       spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i
    22.       expect {
    23.         "yes/no" { send "yes\n";exp_continue }
    24.         "password" { send "1\n" }
    25.       }
    26.       expect "password" { send "1\n" }
    27. EOF
    28. done
    29.  
    30. #2、准备配置文件
    31.  
    32.  
    33.  
    34. # 拼接IP字符串
    35. ip_str=$(printf '", "%s' "${hosts[@]}")
    36. ip_str=${ip_str:3}
    37.  
    38. # 替换文件中的IP地址
    39. sed -i 's#^discovery.*#discovery.seed_hosts: \['"$ip_str"'"\]#g' elasticsearch.yml
    40.  
    41. sed -i 's#^cluster.initial_master_nodes.*#cluster.initial_master_nodes: \['"$ip_str"'"\]#g' elasticsearch.yml
    42.  
    43.  
    44. #3、搭建ES集群
    45.  
    46. for q in ${hosts[*]}
    47. do
    48. 	scp -r elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz elasticsearch.yml es7.service jvm.options jdk.sh $q:/root/
    49. 	ssh $q "mkdir -p /koten/{softwares,data,logs}"
    50. 	ssh $q "tar xf /root/elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz -C /koten/softwares/"
    51. 	ssh $q "useradd koten"
    52. 	ssh $q "chown koten:koten /koten/softwares/elasticsearch-7.17.5/ -R"
    53. 	ssh $q "cat > /etc/security/limits.d/es.conf <
    54. *          soft    nofile     65535
    55. *          hard    nofile     131070
    56. EOF
    57. "	
    58. 	ssh $q "cat > /etc/sysctl.d/es.conf <
    59. vm.max_map_count=262144
    60. EOF"
    61. 	ssh $q "sysctl -f /etc/sysctl.d/es.conf"
    62. 	ssh $q "tar xf /root/jdk-8u291-linux-x64.tar.gz -C /koten/softwares/"
    63. 	ssh $q "\cp /root/elasticsearch.yml  /koten/softwares/elasticsearch-7.17.5/config/"
    64. 	ssh $q "\cp /root/es7.service /usr/lib/systemd/system/"
    65. 	ssh $q "\cp /root/jvm.options /koten/softwares/elasticsearch-7.17.5/config/"
    66. 	ssh $q "\cp /root/jdk.sh /etc/profile.d/"
    67. 	ssh $q "source  /etc/profile.d/jdk.sh"
    68. 	ssh $q "install -d /koten/{data,logs}/es7 -o koten -g koten"
    69. done
    70.  
    71. #3、都部署好后再启动，防止脑裂
    72. for e in ${hosts[*]}
    73. do
    74. 	ssh $e "systemctl enable --now es7"
    75. 	if [ `ss -ntl | grep 19|wc -l` != 0 ];then
    76. 		echo "$e 主机ES启动成功！"
    77. 	fi	
    78. done
    79.  
    80. [root@Ansible ~]# sh deploy_ES.sh
    81. ......
    82. Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
    83. Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
    84. Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
    85.  
    86. [root@ELK104 ~]# netstat -tnulp|grep 19
    87. tcp6       0      0 :::19200                :::*                    LISTEN      2252/java           
    88. tcp6       0      0 :::19300                :::*                    LISTEN      2252/java

     

    Ansible批量部署

    1、初始化roles目录

    1. [root@Ansible roles]# ansible-galaxy init es
    2. - Role es was created successfully
    3. [root@Ansible roles]# cd es 
    4. [root@Ansible es]# pwd
    5. /ansible/roles/es
    6. [root@Ansible es]# ls
    7. defaults  files  handlers  meta  README.md  tasks  templates  tests  vars

    2、编写给10.0.0.104、10.0.0.105、10.0.0.106做免密钥的playbook

    1. [root@Ansible es]# cat /etc/ansible/hosts | tail -4 
    2. [es_group]
    3. es104 ansible_ssh_host=10.0.0.104 ansible_ssh_user='root' ansible_ssh_pass='1'
    4. es105 ansible_ssh_host=10.0.0.105 ansible_ssh_user='root' ansible_ssh_pass='1'
    5. es106 ansible_ssh_host=10.0.0.106 ansible_ssh_user='root' ansible_ssh_pass='1'
    6.  
    7. [root@Ansible es]# cat ssh.yaml 
    8. ---
    9. - hosts: es_group
    10.   gather_facts: no
    11.  
    12.   tasks:
    13.   - name: install ssh key
    14.     authorized_key: user=root
    15.                     key="{{ lookup('file','/root/.ssh/id_rsa.pub')}}"
    16.                     state=present
    17.  
    18. [root@Ansible es]# ansible-playbook ssh.yaml 
    19.  
    20. PLAY [es_group] ****************************************************************************************
    21.  
    22. TASK [install ssh key] *********************************************************************************
    23. changed: [es106]
    24. changed: [es105]
    25. changed: [es104]
    26.  
    27. PLAY RECAP *********************************************************************************************
    28. es104                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
    29. es105                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
    30. es106                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

    3、编写数据推送和配置的playbook

    如果需要修改主机，不仅要修改hosts文件，还需要修改elasticsearch.yml

    1. [root@Ansible roles]# cat hosts | tail -4
    2. [es_group]
    3. 10.0.0.104
    4. 10.0.0.105
    5. 10.0.0.106
    6.  
    7. [root@Ansible roles]# cat site.yml
    8. - hosts: es_group
    9.   roles:
    10.     - role: es
    11.  
    12. [root@Ansible roles]# ls es/files/
    13. elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz  es7.service                 jdk.sh
    14. elasticsearch.yml                                jdk-8u291-linux-x64.tar.gz  jvm.options
    15.  
    16. [root@Ansible roles]# cat es/tasks/main.yml 
    17. - name: mkdir /koten/softwares /koten/data /koten/logs
    18.   file: 
    19.     path: "{{ item }}"
    20.     state: directory
    21.   with_items:
    22.     - /koten/softwares
    23.     - /koten/data
    24.     - /koten/logs
    25.  
    26. - name: tar xf elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz
    27.   unarchive: 
    28.     src: "{{ item }}"
    29.     dest: /koten/softwares
    30.   with_items:
    31.     - elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz
    32.     - jdk-8u291-linux-x64.tar.gz
    33.  
    34. - name: Create koten Group
    35.   group:
    36.     name: koten
    37.  
    38. - name: Create koten User
    39.   user:
    40.     name: koten
    41.  
    42. - name: chown -R koten.koten /koten/softwares/elasticsearch-7.17.5/
    43.   file:
    44.     path: /koten/softwares/elasticsearch-7.17.5/
    45.     owner: koten
    46.     group: koten
    47.     recurse: yes
    48.  
    49. - name: copy content to /etc/security/limits.d/es.conf
    50.   copy:
    51.     content: | 
    52.       *          soft    nofile     65535
    53.       *          hard    nofile     131070
    54.     dest: /etc/security/limits.d/es.conf
    55.  
    56. - name: copy test content to /etc/sysctl.d/es.conf
    57.   copy:
    58.     content: |
    59.       vm.max_map_count=262144
    60.     dest: /etc/sysctl.d/es.conf
    61.  
    62. - name: sysctl -f /etc/sysctl.d/es.conf
    63.   command: sysctl -f /etc/sysctl.d/es.conf
    64.  
    65. - name: push es7.service to /usr/lib/systemd/system/
    66.   copy: 
    67.     src: es7.service
    68.     dest: /usr/lib/systemd/system/
    69.  
    70. - name: push jvm.options to /koten/softwares/elasticsearch-7.17.5/config/
    71.   copy:
    72.     src: jvm.options
    73.     dest: /koten/softwares/elasticsearch-7.17.5/config/
    74.  
    75. - name: push elasticsearch.yml to /koten/softwares/elasticsearch-7.17.5/config/
    76.   copy:
    77.     src: elasticsearch.yml
    78.     dest: /koten/softwares/elasticsearch-7.17.5/config/elasticsearch.yml
    79.  
    80. - name: push jdk.sh to /etc/profile.d/
    81.   copy:
    82.     src: jdk.sh
    83.     dest: /etc/profile.d/
    84.  
    85. - name: source /etc/profile.d/jdk.sh
    86.   command: bash -c 'source /etc/profile.d/jdk.sh'
    87.  
    88. - name: mkdir /koten/data/es7 /koten/logs/es7
    89.   file: 
    90.     path: "{{ item }}"
    91.     state: directory
    92.     owner: koten
    93.     group: koten
    94.   with_items:
    95.     - /koten/data/es7
    96.     - /koten/logs/es7
    97.  
    98. - name: systemctl enable --now es7
    99.   systemd:
    100.     name: es7
    101.     state: started
    102.     enabled: yes

    4、执行数据推送和配置的playbook，部署es集群

    1. [root@Ansible roles]# ansible-playbook site.yml 
    2.  
    3. PLAY [es_group] ****************************************************************
    4.  
    5. TASK [Gathering Facts] *********************************************************
    6. ok: [es105]
    7. ok: [es106]
    8. ok: [es104]
    9.  
    10. TASK [mkdir /koten/softwares /koten/data /koten/logs] **************************
    11. changed: [es105] => (item=/koten/softwares)
    12. changed: [es104] => (item=/koten/softwares)
    13. changed: [es106] => (item=/koten/softwares)
    14. changed: [es106] => (item=/koten/data)
    15. changed: [es104] => (item=/koten/data)
    16. changed: [es105] => (item=/koten/data)
    17. changed: [es106] => (item=/koten/logs)
    18. changed: [es105] => (item=/koten/logs)
    19. changed: [es104] => (item=/koten/logs)
    20.  
    21. TASK [es : tar xf elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz] ***
    22. changed: [es105] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
    23. changed: [es104] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
    24. changed: [es106] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
    25. changed: [es105] => (item=jdk-8u291-linux-x64.tar.gz)
    26. changed: [es106] => (item=jdk-8u291-linux-x64.tar.gz)
    27. changed: [es104] => (item=jdk-8u291-linux-x64.tar.gz)
    28.  
    29. TASK [es : Create koten Group] *************************************************
    30. changed: [es105]
    31. changed: [es106]
    32. changed: [es104]
    33.  
    34. TASK [es : Create koten User] **************************************************
    35. changed: [es104]
    36. changed: [es105]
    37. changed: [es106]
    38.  
    39. TASK [chown -R koten.koten /koten/softwares/elasticsearch-7.17.5/] *************
    40. changed: [es104]
    41. changed: [es105]
    42. changed: [es106]
    43.  
    44. TASK [copy content to /etc/security/limits.d/es.conf] **************************
    45. changed: [es104]
    46. changed: [es106]
    47. changed: [es105]
    48.  
    49. TASK [copy test content to /etc/sysctl.d/es.conf] ******************************
    50. changed: [es106]
    51. changed: [es104]
    52. changed: [es105]
    53.  
    54. TASK [sysctl -f /etc/sysctl.d/es.conf] *****************************************
    55. changed: [es106]
    56. changed: [es104]
    57. changed: [es105]
    58.  
    59. TASK [push es7.service to /usr/lib/systemd/system/] ****************************
    60. changed: [es104]
    61. changed: [es105]
    62. changed: [es106]
    63.  
    64. TASK [push jvm.options to /koten/softwares/elasticsearch-7.17.5/config/] *******
    65. changed: [es104]
    66. changed: [es105]
    67. changed: [es106]
    68.  
    69. TASK [push elasticsearch.yml to /koten/softwares/elasticsearch-7.17.5/config/] ***
    70. changed: [es104]
    71. changed: [es106]
    72. changed: [es105]
    73.  
    74. TASK [es : push jdk.sh to /etc/profile.d/] *************************************
    75. changed: [es104]
    76. changed: [es105]
    77. changed: [es106]
    78.  
    79. TASK [es : source /etc/profile.d/jdk.sh] ***************************************
    80. changed: [es104]
    81. changed: [es106]
    82. changed: [es105]
    83.  
    84. TASK [mkdir /koten/data/es7 /koten/logs/es7] ***********************************
    85. changed: [es104] => (item=/koten/data/es7)
    86. changed: [es105] => (item=/koten/data/es7)
    87. changed: [es106] => (item=/koten/data/es7)
    88. changed: [es105] => (item=/koten/logs/es7)
    89. changed: [es104] => (item=/koten/logs/es7)
    90. changed: [es106] => (item=/koten/logs/es7)
    91.  
    92. TASK [systemctl enable --now es7] **********************************************
    93. changed: [es106]
    94. changed: [es104]
    95. changed: [es105]
    96.  
    97. PLAY RECAP *********************************************************************
    98. es104                      : ok=16   changed=15   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
    99. es105                      : ok=16   changed=15   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
    100. es106                      : ok=16   changed=15   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
    101.  
    102.  
    103.  
    104. # 查看es的状态
    105. [root@ELK104 ~]# systemctl status es7.service 
    106. ● es7.service - es7
    107.    Loaded: loaded (/usr/lib/systemd/system/es7.service; enabled; vendor preset: disabled)
    108.    Active: active (running) since Fri 2023-10-27 14:01:54 CST; 42s ago
    109.  Main PID: 3578 (java)
    110.    CGroup: /system.slice/es7.service
    111.            ├─3578 /koten/softwares/jdk1.8.0_291/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=6...
    112.            └─3711 /koten/softwares/elasticsearch-7.17.5/modules/x-pack-ml/platform/linux-x86_64/bin/c...
    113.  
    114. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...wn]
    115. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...ql]
    116. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...ck]
    117. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,717][INFO ][o.e.p.PluginsService ...re]
    118. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,717][INFO ][o.e.p.PluginsService ...de]
    119. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,749][INFO ][o.e.p.PluginsService ...er]
    120. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,750][INFO ][o.e.p.PluginsService ...ded
    121. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,958][INFO ][o.e.e.NodeEnvironment...fs]
    122. Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,959][INFO ][o.e.e.NodeEnvironment...ue]
    123. Oct 27 14:02:20 ELK104 elasticsearch[3578]: [2023-10-27T14:02:20,074][INFO ][o.e.n.Node           ...st]
    124. Hint: Some lines were ellipsized, use -l to show in full.

     

    deploy_es_shell打包文件下载链接：https://pan.baidu.com/s/1l22I0aSB-50NXlh4lYZArw?pwd=umbm

    deploy_es_ansible打包文件下载链接：https://pan.baidu.com/s/1lIfYWMnhsg7yoyzgkBSEDA?pwd=8hpl

    我是koten，10年运维经验，持续分享运维干货，感谢大家的阅读和关注！

  • 相关阅读:
    五步走，轻松拥有你的个性化小程序
    Python图像处理丨三种实现图像形态学转化运算模式
    【DS基础】栈与队列
    FISCO BCOS(三十七)———FISCOBCOS应用开发，交易hash、区块高度的获取
    SpringMVC中如何编写一个Controller呢？
    ffmpeg音频重采样
    微信小程序OA会议系统个人中心授权登入
    MarkDown实现文章内跳转
    【Linux-网络编程】
    java笔试题含答案总结四
  • 原文地址：https://blog.csdn.net/qq_37510195/article/details/130855070