• Nginx + Keepalived 高可用集群部署

    正文

    负载均衡技术对于一个网站尤其是大型网站的web服务器集群来说是至关重要的！做好负载均衡架构，可以实现故障转移和高可用环境，避免单点故障，保证网站健康持续运行。在使用 Nginx 做反向代理或者负载均衡的时候，都是以 Nginx 为入口，如果 Nginx 宕机了，那么所有的服务都无法正常提供，影响非常严重。

    为了避免负载均衡服务器宕机造成严重影响，就需要建立一个备份机。主服务器和备份机上都运行高可用（High Availability）监控程序，通过传送诸如“I am alive”这样的信息来监控对方的运行状况。当备份机不能在一定的时间内收到这样的信息时，它就接管主服务器的服务IP并继续提供负载均衡服务；当备份管理器又从主管理器收到“I am alive”这样的信息时，它就释放服务IP地址，这样的主服务器就开始再次提供负载均衡服务。

    高可用（High Availability）是分布式系统架构设计中必须考虑的因素之一，它通常是指，通过设计减少系统不能提供服务的时间。如果一个系统能够一直提供服务，那么这个可用性则是百分之百，但是我们不能保证一个系统能永远不出问题，所以我们只能通过设计来尽可能的去减少由于系统的故障所造成的影响。

    由于业务扩展，网站的访问量不断加大，负载越来越高。现需要在web前端放置nginx负载均衡,同时结合keepalived对前端nginx实现HA高可用。

    前文分享了《Linux下Nginx基础应用》，《Linux下实现高可用软件-Keepalived基础知识梳理》；今天简单分享Nginx + Keepalived 高可用集群部署。

    主从集群架构图

    环境说明
    hostname ip 说明
    Client-01 172.16.70.171 客户端测试机
    KeepMaster 172.16.70.181 keepalived 主服务器 (nginx 主负载均衡器)
    KeepBackup 172.16.70.182 keepalived 备服务器 (nginx 备负载均衡器)
    VIP 172.16.70.183 vrrp HA 虚拟地址，可有多个IP
    Web1 172.16.70.191 后端web主服务器1 (nginx 站点)
    Web2 172.16.70.192 后端web备服务器2 (nginx 站点)

    准备环境

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    # 本次所有部署服务器都配置
    # cat /etc/redhat-release
    CentOS Linux release 7.9.2009 (Core)
    # uname -r
    3.10.0-1160.83.1.el7.x86_64
     
    # systemctl stop firewalld
    # sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/sysconfig/selinux
    # setenforce 0
    # ntpdate 0.centos.pool.ntp.org
    # yum install net-tools vim wget curl -y

     搭建后端web服务器

    • 主备一样操作
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    # 这里以 web-01 为例
    [root@web-01 ~]# wget https://nginx.org/packages/centos/7/x86_64/RPMS/nginx-1.20.2-1.el7.ngx.x86_64.rpm
    [root@web-01 ~]# rpm -ivh nginx-1.20.2-1.el7.ngx.x86_64.rpm
    [root@web-01 ~]# nginx -v
    nginx version: nginx/1.20.2
     
    [root@web-01 ~]# #echo "`hostname` `ifconfig ens33 |sed -n 's#.*inet \(.*\)netmask.*#\1#p'`" > /usr/share/nginx/html/index.html
    [root@web-01 ~]# cat /usr/share/nginx/html/index.html
    web-01 172.16.70.191
     
    [root@web-01 ~]# nginx -t
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
     
    [root@web-01 ~]# systemctl start nginx
    [root@web-01 ~]# systemctl enable nginx
    [root@web-01 ~]# netstat -ntupl | grep nginx
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      10687/nginx: master
    [root@web-01 ~]# ps -ef | grep nginx
    root      10687      1  0 16:36 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    nginx     10688  10687  0 16:36 ?        00:00:00 nginx: worker process
    nginx     10689  10687  0 16:36 ?        00:00:00 nginx: worker process
    nginx     10690  10687  0 16:36 ?        00:00:00 nginx: worker process
    nginx     10691  10687  0 16:36 ?        00:00:00 nginx: worker process
    root      10761  10586  0 16:45 pts/1    00:00:00 grep --color=auto nginx
      
    # Client-01 测试访问
    [root@Client-01 ~]# curl 172.16.70.191
    web-01 172.16.70.191

     浏览器测试访问 http://ip/

     

    Keep服务器上部署nginx负载均衡器

    • 主备一样操作
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    # 这里以 KeepMaster 为例
    # 安装部署nginx
    [root@KeepMaster ~]# wget https://nginx.org/packages/centos/7/x86_64/RPMS/nginx-1.18.0-2.el7.ngx.x86_64.rpm
    [root@KeepMaster ~]# rpm -vih nginx-1.18.0-2.el7.ngx.x86_64.rpm
    [root@KeepMaster ~]# nginx -v
    nginx version: nginx/1.18.0
     
    # 新建
    [root@KeepMaster ~]# cat /etc/nginx/conf.d/web.conf
    upstream web {
        server 172.16.70.191:80 weight=1 max_fails=3 fail_timeout=20s;
        server 172.16.70.192:80 weight=2 max_fails=3 fail_timeout=20s;
    }
        # weight(权重)和访问比率成正比，默认值为1
        # max_fails 为允许失败的次数，默认值为1
        # fail_timeout 当max_fails次失败后，暂停将请求分发到该后端服务器的时间
     
    server {
      listen 80;
      server_name www.zhangwencheng.org;
       
      location / {
        proxy_pass http://web;
        proxy_set_header HOST $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
    }
     
    [root@KeepMaster ~]# nginx -t
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
     
    [root@KeepMaster ~]# systemctl start nginx
    [root@KeepMaster ~]# systemctl enable nginx
     
    [root@KeepMaster ~]# ps -ef | grep nginx
    root       1677      1  0 17:28 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    nginx      1678   1677  0 17:28 ?        00:00:00 nginx: worker process
    root       1708   1444  0 17:33 pts/0    00:00:00 grep --color=auto nginx
    [root@KeepMaster ~]# netstat -tnpl | grep nginx
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1677/nginx: master

    测试机Client-01验证负载均衡

    • Keep主备服务器上的nginx负载均衡
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    # 在测试机上添加host解析, KeepMaster/KeepBackup主机IP
    [root@Client-01 ~]# cat /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    172.16.70.181   www.zhangwencheng.org
    172.16.70.182   www.zhangwencheng.org
     
    # 测试时候轮流关闭Keep节点，关闭后还是能够访问并看到轮循效果即表示nginx负载均衡器集群搭建成功。
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-01 172.16.70.191 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-01 172.16.70.191

    Keep服务器上部署keepalived

    • 主备一样操作
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    [root@KeepMaster ~]# yum install -y openssl openssl-devel libnl libnl-devel gcc
    [root@KeepMaster ~]# mkdir /data/apps/keepalived -p
    [root@KeepMaster ~]# wget --no-check-certificate  http://www.keepalived.org/software/keepalived-2.2.4.tar.gz
    [root@KeepMaster ~]# tar -xf keepalived-2.2.4.tar.gz
    [root@KeepMaster ~]# cd keepalived-2.2.4
    [root@KeepMaster keepalived-2.2.4]# ls
    aclocal.m4  autogen.sh   build-aux    ChangeLog  configure.ac  COPYING  INSTALL     keepalived.spec.in  m4           Makefile.in  snap  tools
    AUTHOR      bin_install  build_setup  configure  CONTRIBUTORS  doc      keepalived  lib                 Makefile.am  README.md    TODO
     
    [root@KeepMaster keepalived-2.2.4]# ./configure --prefix=/data/apps/keepalived
    ....
    ....
    # 最后编译正常输出如下
    Keepalived configuration
    ------------------------
    Keepalived version       : 2.2.4
    Compiler                 : gcc gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
    Preprocessor flags       : -D_GNU_SOURCE
    Compiler flags           : -g -g -O2 -Wextra -Wunused -Wstrict-prototypes -Wabi -Wbad-function-cast -Wcast-align -Wcast-qual -Wdisabled-optimization -Wdouble-promotion \
    -Wfloat-equal -Wframe-larger-than=5120 -Winit-self -Winline -Winvalid-pch -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-field-initializers -Wmissing-include-dirs \
    -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Woverlength-strings -Wpointer-arith -Wredundant-decls -Wshadow -Wstack-protector -Wstrict-overflow=4 -Wsuggest-attribute=format \
    -Wsuggest-attribute=noreturn -Wsuggest-attribute=pure -Wsync-nand -Wtrampolines -Wundef -Wuninitialized -Wunknown-pragmas -Wunsafe-loop-optimizations -Wunsuffixed-float-constants -Wvariadic-macros \
    -Wwrite-strings -fPIE -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -O2
    Linker flags             : -pie -Wl,-z,relro -Wl,-z,now
    Extra Lib                : -lm -lcrypto -lssl -lnl
    Use IPVS Framework       : Yes
    IPVS use libnl           : Yes
    IPVS syncd attributes    : No
    IPVS 64 bit stats        : No
    HTTP_GET regex support   : No
    fwmark socket support    : Yes
    Use VRRP Framework       : Yes
    Use VRRP VMAC            : Yes
    Use VRRP authentication  : Yes
    With track_process       : Yes
    With linkbeat            : Yes
    Use BFD Framework        : No
    SNMP vrrp support        : No
    SNMP checker support     : No
    SNMP RFCv2 support       : No
    SNMP RFCv3 support       : No
    DBUS support             : No
    Use JSON output          : No
    libnl version            : 1
    Use IPv4 devconf         : No
    Use iptables             : No
    Use nftables             : No
    init type                : systemd
    systemd notify           : No
    Strict config checks     : No
    Build documentation      : No
    Default runtime options  : -D
     
    [root@KeepMaster keepalived-2.2.4]# make -j 4 && make install
    [root@KeepMaster keepalived-2.2.4]# ls
    aclocal.m4  bin          build_setup  config.status  CONTRIBUTORS  INSTALL          keepalived.spec.in  Makefile     README     TODO
    AUTHOR      bin_install  ChangeLog    configure      COPYING       keepalived       lib                 Makefile.am  README.md  tools
    autogen.sh  build-aux    config.log   configure.ac   doc           keepalived.spec  m4                  Makefile.in  snap
     
    [root@KeepMaster keepalived-2.2.4]# cp keepalived/keepalived /usr/local/sbin/ -a
    [root@KeepMaster keepalived-2.2.4]# keepalived -v
    Keepalived v2.2.4 (08/21,2021)
     
    Copyright(C) 2001-2021 Alexandre Cassen,
     
    Built with kernel headers for Linux 3.10.0
    Running on Linux 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023
    Distro: CentOS Linux 7 (Core)
     
    configure options: --prefix=/data/apps/keepalived
     
    Config options:  LVS VRRP VRRP_AUTH VRRP_VMAC OLD_CHKSUM_COMPAT INIT=systemd
     
    System options:  VSYSLOG LIBNL1 RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA IFA_FLAGS \
    NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION LIBIPVS_NETLINK IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE SO_MARK
     
    [root@KeepMaster keepalived-2.2.4]# cd /data/apps/keepalived/
    [root@KeepMaster keepalived]# ls
    bin  etc  sbin  share
     
    [root@KeepMaster keepalived]# mv etc/keepalived/keepalived.conf etc/keepalived/keepalived.conf_bak
    [root@KeepMaster keepalived]# vim etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
     
    global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 192.168.200.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
     
    vrrp_script chk_nginx {    
        script "/data/apps/keepalived/chk_nginx.sh" 
        interval 2                 
        weight -5                                  
    }
     
    vrrp_instance VI_1 {
        state MASTER　　　　# 备服务器这为 BACKUP
        interface ens33
        virtual_router_id 51
        priority 110　　　　# 备服务器这小于110
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
           172.16.70.183
        }
    track_script {
       chk_nginx
        }
    }

    优先级不会不断的提高或者降低，最终优先级的范围是在[1,254]，不会出现优先级小于等于0或者优先级大于等于255的情况。
    在MASTER节点的vrrp_instance中配置nopreempt，当它异常恢复后，即使它prio更高也不会抢占，这样可以避免正常情况下做无谓的切换。

    nginx检测脚本

    编写脚本来判断本机nginx是否正常，如果发现NginX不正常，自重启nginx。等待2秒再次校验，仍然失败则不再尝试，关闭keepalived，让其他主机此时会接管VIP。
    此脚本必须在keepalived服务运行的前提下才有效！如果在keepalived服务先关闭的情况下，那么nginx服务关闭后就不能实现自启动了。

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    [root@KeepMaster keepalived]# cat chk_nginx.sh
    #!/bin/bash
    chk=$(ps -C nginx --no-heading|wc -l)
    if [ "${chk}" = "0" ]; then
        systemctl start nginx
        sleep 2
        chk=$(ps -C nginx --no-heading|wc -l)
        if [ "${chk}" = "0" ]; then
            systemctl stop keepalived
        fi
    fi
     
    [root@KeepMaster keepalived]# chmod +x chk_nginx.sh

    测试机Client-01验证VIP

    • Keep服务器上的VIP
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    [root@Client-01 ~]# cat /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    172.16.70.183   www.zhangwencheng.org
     
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-01 172.16.70.191 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192

    故障转移测试

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    #手动关闭Master机器上的nginx服务，最多2秒钟后就会自启动
    [root@KeepMaster ~]# systemctl stop nginx
    [root@KeepMaster ~]# ps -ef | egrep 'nginx|keepalived'
    root      57266      1  0 16:21 ?        00:00:00 /data/apps/keepalived/sbin/keepalived -f /data/apps/keepalived/etc/keepalived/keepalived.conf -D
    root      57267  57266  0 16:21 ?        00:00:01 /data/apps/keepalived/sbin/keepalived -f /data/apps/keepalived/etc/keepalived/keepalived.conf -D
    root      60019      1  0 16:42 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    nginx     60020  60019  0 16:42 ?        00:00:00 nginx: worker process
    root      60027   1444  0 16:42 pts/0    00:00:00 grep -E --color=auto nginx|keepalived
    [root@KeepMaster ~]# ip a
    1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 00:0c:29:a1:82:4e brd ff:ff:ff:ff:ff:ff
        inet 172.16.70.181/24 brd 172.16.70.255 scope global noprefixroute ens33
           valid_lft forever preferred_lft forever
        inet 172.16.70.183/32 scope global ens33
           valid_lft forever preferred_lft forever
        inet6 fe80::7726:d409:2cf4:babd/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::833:43b:7d2:6e4c/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::c2be:590b:1ae6:42e3/64 scope link noprefixroute
           valid_lft forever preferred_lft forever
     
    #手动关闭Master机器上的keepalived服务，已经发现没VIP了
    [root@KeepMaster ~]# systemctl stop keepalived
    [root@KeepMaster ~]# ps -ef | egrep 'nginx|keepalived'
    root      60019      1  0 16:42 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    nginx     60020  60019  0 16:42 ?        00:00:00 nginx: worker process
    root      60348   1444  0 16:45 pts/0    00:00:00 grep -E --color=auto nginx|keepalived
    [root@KeepMaster ~]# ip a
    1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 00:0c:29:a1:82:4e brd ff:ff:ff:ff:ff:ff
        inet 172.16.70.181/24 brd 172.16.70.255 scope global noprefixroute ens33
           valid_lft forever preferred_lft forever
        inet6 fe80::7726:d409:2cf4:babd/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::833:43b:7d2:6e4c/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::c2be:590b:1ae6:42e3/64 scope link noprefixroute
           valid_lft forever preferred_lft forever
     
    [root@KeepMaster keepalived]# tail /var/log/messages
    Mar  7 17:01:01 Keepalived-01 systemd: Started Session 27 of user root.
    Mar  7 17:01:28 Keepalived-01 Keepalived[60703]: Stopping
    Mar  7 17:01:28 Keepalived-01 systemd: Stopping LVS and VRRP High Availability Monitor...
    Mar  7 17:01:28 Keepalived-01 Keepalived_vrrp[60704]: (VI_1) sent 0 priority
    Mar  7 17:01:28 Keepalived-01 Keepalived_vrrp[60704]: (VI_1) removing VIPs.
    Mar  7 17:01:28 Keepalived-01 NetworkManager[570]:   [1678179688.0176] policy: set-hostname: current hostname was changed outside NetworkManager: 'KeepMaster'
    Mar  7 17:01:29 Keepalived-01 Keepalived_vrrp[60704]: Stopped - used (self/children) 0.005506/0.797936 user time, 0.172766/0.818969 system time
    Mar  7 17:01:29 Keepalived-01 Keepalived[60703]: CPU usage (self/children) user: 0.000000/0.803442 system: 0.001394/0.994146
    Mar  7 17:01:29 Keepalived-01 Keepalived[60703]: Stopped Keepalived v2.2.4 (08/21,2021)
    Mar  7 17:01:29 Keepalived-01 systemd: Stopped LVS and VRRP High Availability Monitor
    • KeepBackup查看，已经接管VIP
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    [root@KeepBackup keepalived]# ps -ef | egrep 'nginx|keepalived'
    root      65036      1  0 17:01 ?        00:00:00 /data/apps/keepalived/sbin/keepalived -f /data/apps/keepalived/etc/keepalived/keepalived.conf -D
    root      65037  65036  0 17:01 ?        00:00:00 /data/apps/keepalived/sbin/keepalived -f /data/apps/keepalived/etc/keepalived/keepalived.conf -D
    root      65067      1  0 17:01 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    nginx     65068  65067  0 17:01 ?        00:00:00 nginx: worker process
    root      65122   1514  0 17:01 pts/0    00:00:00 grep -E --color=auto nginx|keepalived
     
    [root@KeepBackup keepalived]# ip a
    1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 00:0c:29:c4:54:23 brd ff:ff:ff:ff:ff:ff
        inet 172.16.70.182/24 brd 172.16.70.255 scope global noprefixroute ens33
           valid_lft forever preferred_lft forever
        inet 172.16.70.183/32 scope global ens33
           valid_lft forever preferred_lft forever
        inet6 fe80::7726:d409:2cf4:babd/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::833:43b:7d2:6e4c/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::c2be:590b:1ae6:42e3/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
    [root@KeepBackup keepalived]# tail /var/log/messages
    Mar  7 17:01:28 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:28 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:28 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:28 Keepalived-02 NetworkManager[573]:   [1678179688.8137] policy: set-hostname: current hostname was changed outside NetworkManager: 'KeepBackup'
    Mar  7 17:01:33 Keepalived-02 Keepalived_vrrp[65037]: (VI_1) Sending/queueing gratuitous ARPs on ens33 for 172.16.70.183
    Mar  7 17:01:33 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:33 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:33 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:33 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
    Mar  7 17:01:33 Keepalived-02 Keepalived_vrrp[65037]: Sending gratuitous ARP on ens33 for 172.16.70.183
     
     
     

    双机高可用一般是通过虚拟IP（飘移IP）方法来实现的，基于Linux/Unix的IP别名技术，目前分为两种：

    • 双机主从模式：即前端使用两台服务器，一台主服务器和一台热备服务器，正常情况下，主服务器绑定一个公网虚拟IP，提供负载均衡服务，热备服务器处于空闲状态；当主服务器发生故障时，热备服务器接管主服务器的公网虚拟IP，提供负载均衡服务；但是热备服务器在主机器不出现故障的时候，永远处于浪费状态，对于服务器不多的网站，该方案不经济实惠。
    • 双机主主模式：即前端使用两台负载均衡服务器，互为主备，且都处于活动状态，同时各自绑定一个公网虚拟IP，提供负载均衡服务；当其中一台发生故障时，另一台接管发生故障服务器的公网虚拟IP（这时由非故障机器一台负担所有的请求）。这种方案，经济实惠，非常适合于当前架构环境。

    主主模式集群架构图

    当了解主备模式后，双主模式就容易配置多了。只需要在每台keepalived配置文件，加上一个vrrp_instance命名vrrp_instance VI_2即可，更改几个参数，设置另一个VIP：172.16.70.184

    • KeepMaster：state BACKUP ，priority 100， virtual_router_id 52
    • KeepBackup：state MASTER ，priority 110， virtual_router_id 52

    KeepMaster上的keepalived.conf

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    [root@KeepMaster keepalived]# cat etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
     
    global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 192.168.200.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
     
    vrrp_script chk_nginx {    
        script "/data/apps/keepalived/chk_nginx.sh" 
        interval 2                 
        weight -5                  
    }
     
    vrrp_instance VI_1 {
        state MASTER
        interface ens33
        virtual_router_id 51
        priority 110
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
           172.16.70.183
        }
    track_script {
       chk_nginx
        }
    }
     
    vrrp_instance VI_2 {
        state BACKUP
        interface ens33
        virtual_router_id 52
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
           172.16.70.184
        }
    track_script {
       chk_nginx
        }
    }
     
    [root@KeepMaster keepalived]# systemctl restart keepalived

     KeepBackup上的keepalived.conf

    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    [root@KeepBackup keepalived]# cat etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
     
    global_defs {
       notification_email {
         acassen@firewall.loc
         failover@firewall.loc
         sysadmin@firewall.loc
       }
       notification_email_from Alexandre.Cassen@firewall.loc
       smtp_server 192.168.200.1
       smtp_connect_timeout 30
       router_id LVS_DEVEL
       vrrp_skip_check_adv_addr
       vrrp_strict
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
     
    vrrp_script chk_nginx {    
        script "/data/apps/keepalived/chk_nginx.sh" 
        interval 2                 
        weight -5                  
    }
     
    vrrp_instance VI_1 {
        state BAKCUP
        interface ens33
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
           172.16.70.183
        }
    track_script {
       chk_nginx
        }
    }
     
    vrrp_instance VI_2 {
        state MASTER
        interface ens33
        virtual_router_id 52
        priority 110
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
           172.16.70.184
        }
    track_script {
       chk_nginx
        }
    }
     
    [root@KeepBackup keepalived]# systemctl restart keepalived

     测试机Client-01验证VIP2

    • Keep服务器上的VIP2
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # 注释VIP1解析，此时仅测试VIP2；测试成功后再取消VIP1注释
    [root@Client-01 ~]# cat /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    #172.16.70.183  www.zhangwencheng.org
    172.16.70.184   www.zhangwencheng.org
     
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-01 172.16.70.191 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-02 172.16.70.192 
    [root@Client-01 ~]# curl www.zhangwencheng.org
    web-01 172.16.70.191
    •  查看VIP情况
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    [root@KeepMaster ~]# ip a
    1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 00:0c:29:a1:82:4e brd ff:ff:ff:ff:ff:ff
        inet 172.16.70.181/24 brd 172.16.70.255 scope global noprefixroute ens33
           valid_lft forever preferred_lft forever
        inet 172.16.70.183/32 scope global ens33　　　　# VIP1
           valid_lft forever preferred_lft forever
        inet6 fe80::7726:d409:2cf4:babd/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::833:43b:7d2:6e4c/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::c2be:590b:1ae6:42e3/64 scope link noprefixroute
           valid_lft forever preferred_lft forever
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    [root@KeepBackup ~]# ip a
    1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens33: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 00:0c:29:c4:54:23 brd ff:ff:ff:ff:ff:ff
        inet 172.16.70.182/24 brd 172.16.70.255 scope global noprefixroute ens33
           valid_lft forever preferred_lft forever
        inet 172.16.70.184/32 scope global ens33　　　　# VIP2
           valid_lft forever preferred_lft forever
        inet6 fe80::7726:d409:2cf4:babd/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::833:43b:7d2:6e4c/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
        inet6 fe80::c2be:590b:1ae6:42e3/64 scope link tentative noprefixroute dadfailed
           valid_lft forever preferred_lft forever
    •  故障转移测试类同，不再赘述。
  • 相关阅读:
    Python-scapy模块的使用
    Spring AOP如何使用AspectJ注解进行开发呢？
    沉睡者IT创业项目 - 学习如何玩直播卖零食
    vi, vim，data，wc，系统常用命令-读书笔记（十）
    R语言绘制高密度散点图
    网络编程：socket的阻塞模式和非阻塞模式
    vue3.0中使用echarts，鼠标悬浮无法显示数据框的问题
    27.CSS粒子特效
    自动化测试开发 —— 如何封装自动化测试框架？
    数据库系列：使用高区分度索引列提升性能
  • 原文地址：https://www.cnblogs.com/zhangwencheng/p/17182896.html