云原生|kubernetes|kubeadm部署高可用集群（二）---kube-apiserver高可用+etcd外部集群

前言：

上一篇文章云原生|kubernetes|kubeadm部署高可用集群（一）使用外部etcd集群_晚风_END的博客-CSDN博客讲述了如何利用kubeadm部署集群的时候使用外部扩展etcd集群，使得集群的整体结构做了一些解耦，提高了集群的负载能力，那么，本文将讲述如何在此基础上做到kube-apiserver的高可用，从而部署一个可完全用于生产的kubernetes集群。

下面就直接上干货。

一，集群环境简介

本次实践计划使用haproxy+keepalived针对kube-apiserver搭建一个负载均衡。为什么不使用nginx+keepalived的技术栈呢？主要是考虑到主机性能不太够，因此虚拟机不够多，其次nginx做负载均衡的时候，会占用集群的默认6443端口，会有一些不必要的麻烦，而haproxy是一个更为专业的代理软件。

kubernetes集群内部使用三个kube-apiserver+外部扩展etcd集群，各个组件都是高可用的，完全可以应用在实际的生产活动中。

kubernetes集群是由三个master节点和一个node工作节点组成，master节点使用三个是出于集群的奇数要求，两个或者四个提现不出高可用的特点。工作节点的扩展是比较简单的，如果在实际生产中可以很简单的就扩展工作节点，因此工作节点设置为一个。

下面是整个集群的大体配置：

**kubernetes高可用集群配置表**
服务器IP地址	集群角色	服务器硬件配置	安装的组件	组件版本	部署方式
192.168.217.19	control-plane,master	CPU：2c2u 内存：4G 硬盘：100G 单分区 /	kubelet，kubeadm，kubernetes集群其它关键组件为静态pod形式。负载均衡：haproxy和keepalived etcd集群 docker环境	kubernetes-1.22.2 docker-ce-20.10.5 haproxy-1.5.18-9 keepalived-1.3.5 etcd Version: 3.4.9	docker和etcd：二进制方式其它：yum安装
192.168.217.20	control-plane,master	CPU：2c2u 内存：4G 硬盘：100G 单分区 /	kubelet，kubeadm，kubernetes集群其它关键组件为静态pod形式。负载均衡：haproxy和keepalived etcd集群 docker环境	kubernetes-1.22.2 docker-ce-20.10.5 haproxy-1.5.18-9 keepalived-1.3.5 etcd Version: 3.4.9	docker和etcd：二进制方式其它：yum安装
192.168.217.21	control-plane,master	CPU：2c2u 内存：4G 硬盘：100G 单分区 /	kubelet，kubeadm，kubernetes集群其它关键组件为静态pod形式。负载均衡：haproxy和keepalived etcd集群 docker环境	kubernetes-1.22.2 docker-ce-20.10.5 haproxy-1.5.18-9 keepalived-1.3.5 etcd Version: 3.4.9	docker和etcd：二进制方式其它：yum安装
192.168.217.22	node，工作节点	CPU：2c2u 内存：4G 硬盘：100G 单分区 /	kubelet，kubeadm，kubernetes集群其它关键组件为静态pod形式。 docker环境	kubernetes-1.22.2 docker-ce-20.10.5 etcd Version: 3.4.9	docker：二进制方式其它：yum安装注意：etcd集群是三节点形式，因此，本节点不安装
VIP：192.168.217.100	负载均衡节点	无	无	keepalived产生


[root@master1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.217.19 master1 k8s-master1
192.168.217.20 master2 k8s-master2
192.168.217.21 master3 k8s-master3
192.168.217.22 node1 k8s-node1

二，

集群基础环境搭建

（1）配置主机名称：

k8s-master1


[root@master1 ~]# cat /etc/hostname 
master1

k8s-master2


[root@master2 manifests]# cat /etc/hostname
master2

k8s-master3


[root@master3 ~]# cat /etc/hostname
master3

node1


[root@node1 ~]# cat /etc/hostname
node1

所有服务器统一hosts：


[root@master1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.217.19 master1 k8s-master1
192.168.217.20 master2 k8s-master2
192.168.217.21 master3 k8s-master3
192.168.217.22 node1 k8s-node1

（2）时间服务器

所有节点安装ntp并启动ntpd服务：


yum install ntp -y
systemctl enable ntpd && systemctl start ntpd

192.168.217.19作为时间服务器，配置文件主要的地方：


# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 127.127.1.0 prefer
fudge 127.127.1.0 stratum 10

其它节点的时间服务器配置：


# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 192.168.217.19

任意一个节点输出如下表示时间服务器正常：


[root@master2 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*master1         LOCAL(0)        11 u   26  128  377    0.390    0.014   0.053
[root@master2 ~]# ntpstat
synchronised to NTP server (192.168.217.19) at stratum 12
   time correct to within 23 ms
   polling server every 128 s

（3）关闭防火墙

systemctl disable firewalld && systemctl stop firewalld

（4）关闭selinux

编辑/etc/selinux/config文件：


[root@master2 ~]# cat /etc/selinux/config 
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

输出如下表示关闭：


[root@master2 ~]# getenforce 
Disabled

（5）关闭swap

这个就不说了，太基础的东西。

（6）服务器之间的免密码

也不在这详细说了，免密的原理和如何免密见我的博客：科普扫盲---ssh免密登陆（ssh的一些小秘密）_晚风_END的博客-CSDN博客_ssh免密不带端口号

这里配置免密的原因是由于后面初始化集群的时候需要服务器自动传输证书。

（7）docker环境的搭建

docker的离线安装以及本地化配置_晚风_END的博客-CSDN博客务必要做好本地化的配置，否则后面的下载镜像会成为噩梦。

部署完成后按教程做好docker环境的测试工作。

（8）etcd集群的搭建

centos7操作系统 ---ansible剧本离线快速部署etcd集群_晚风_END的博客-CSDN博客_etcd离线安装务必按此教程部署etcd集群，后面的集群初始化工作会基于此etcd集群开始。

部署完成后，按教程做好etcd集群的测试工作。

（9）

安装kubelet，kubectl，kubeadm

云原生|kubernetes|kubeadm五分钟内部署完成集群（完全离线部署---适用于centos7全系列）_晚风_END的博客-CSDN博客

其实集群搭建也就麻烦在基础环境的搭建，费时费力，不过有一个良好的开端的话，后面会比较轻松一些。

三，创建负载均衡器（HAProxy+Keepalived）

当存在多个控制平面时，kube-apiserver也存在多个，可以使用Nginx+Keepalived、HAProxy+Keepalived等工具实现多个kube-apiserver的负载均衡和高可用。
推荐使用HAProxy+Keepalived这个组合，因为HAProxy可以提高更高性能的四层负载均衡功能，这也是大多数人的选择。

负载均衡架构图：

在三个master节点都安装haproxy+keepalived：

yum install haproxy keepalived -y

haproxy的配置（绑定9443端口，监听三个apiserver并组成后端，此配置文件复制到其它两个master节点）：


cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
 
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
 
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
 
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
 
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
 
frontend apiserver
bind *:9443
mode tcp
option tcplog
default_backend apiserver
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend apiserver
    balance     roundrobin
    server k8s-master1 192.168.217.19:6443 check
    server k8s-master2 192.168.217.20:6443 check
    server k8s-master3 192.168.217.21:6443 check
 
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
 
listen admin_stats  
    bind 0.0.0.0:9188  #登录页面所绑定的地址加端口
    mode http #监控的模式
    log 127.0.0.1 local0 err #错误日志等级
    stats refresh 30s
    stats uri /haproxy-status   #登录页面的网址，IP:9188/haproxy-status 即为登录网址
    stats realm welcome login\ Haproxy 
    stats auth admin:admin123       #web页面的用户名和密码
    stats hide-version 
    stats admin if TRUE

复制haproxy配置文件：


scp /etc/haproxy/haproxy.cfg master2:/etc/haproxy/
scp /etc/haproxy/haproxy.cfg master3:/etc/haproxy/

打开任意浏览器，输入ip+9188，可以打开haproxy的web界面（账号：admin 密码：admin123）：

keepalived的配置：

master1节点的keepalived的配置文件：


[root@master1 ~]# cat /etc/keepalived/keepalived.conf 
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 192.168.217.19
}
vrrp_script check_haproxy {
script "bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi'"
interval 3
weight -2
fall 3
rise 3
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 50
priority 100
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.217.100
}
track_script {
check_haproxy
}
}

master节点的keepalived的配置文件：


[root@master2 ~]# cat /etc/keepalived/keepalived.conf 
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 192.168.217.20
}
vrrp_script check_haproxy {
script "bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi'"
interval 3
weight -2
fall 3
rise 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 50
priority 99
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.217.100
}
track_script {
check_haproxy
}
}

master3节点的keepalived的配置文件：


[root@master3 ~]# cat /etc/keepalived/keepalived.conf 
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id 192.168.217.21
}
vrrp_script check_haproxy {
script "bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi'"
interval 3
weight -2
fall 3
rise 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 50
priority 98
authentication {
auth_type PASS
auth_pass 123456
}
virtual_ipaddress {
192.168.217.100
}
track_script {
check_haproxy
}
}

两个服务的自启和启动：

systemctl enable keepalived haproxy && systemctl start keepalived haproxy

负载均衡的测试：

查看端口是否开启：


[root@master1 ~]# netstat -antup |grep 9443
tcp        0      0 0.0.0.0:9443            0.0.0.0:*               LISTEN      1011/haproxy

在master1上查看vip（为什么在master1呢？因为它的优先级是100）：


[root@master1 ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:5c:1e:66 brd ff:ff:ff:ff:ff:ff
    inet 192.168.217.19/24 brd 192.168.217.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.217.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe5c:1e66/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0:  mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:f3:9c:2f:92 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

关闭master1的haproxy服务，查看vip是否漂移到master2：

[root@master1 ~]# systemctl stop haproxy

在master2上查看：


[root@master2 ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:86:30:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.217.20/24 brd 192.168.217.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.217.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe86:30e6/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0:  mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:8a:27:b6:7f brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

漂移成功，这一步一定要成功才可以进行下一步。

在master1上恢复haproxy服务，可以看到很迅速的vip就回来了：


[root@master1 ~]# systemctl start haproxy
[root@master1 ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:5c:1e:66 brd ff:ff:ff:ff:ff:ff
    inet 192.168.217.19/24 brd 192.168.217.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.217.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe5c:1e66/64 scope link 
       valid_lft forever preferred_lft forever

查看master1的系统日志，看看vip漂移的过程：


Oct 27 19:01:28 master1 Keepalived_vrrp[1037]: /usr/bin/bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi' exited with status 1
Oct 27 19:01:31 master1 Keepalived_vrrp[1037]: /usr/bin/bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi' exited with status 1
Oct 27 19:01:34 master1 Keepalived_vrrp[1037]: /usr/bin/bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi' exited with status 1
Oct 27 19:01:37 master1 Keepalived_vrrp[1037]: /usr/bin/bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi' exited with status 1
Oct 27 19:01:38 master1 systemd: Started HAProxy Load Balancer.
Oct 27 19:01:38 master1 systemd: Starting HAProxy Load Balancer...
Oct 27 19:01:38 master1 haproxy-systemd-wrapper: [WARNING] 299/190138 (31426) : config : 'option forwardfor' ignored for frontend 'apiserver' as it requires HTTP mode.
Oct 27 19:01:46 master1 Keepalived_vrrp[1037]: VRRP_Script(check_haproxy) succeeded
Oct 27 19:01:47 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Changing effective priority from 98 to 100
Oct 27 19:01:47 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) forcing a new MASTER election
Oct 27 19:01:48 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Transition to MASTER STATE
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Entering MASTER STATE
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) setting protocol VIPs.
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: Sending gratuitous ARP on ens33 for 192.168.217.100
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.217.100
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: Sending gratuitous ARP on ens33 for 192.168.217.100
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: Sending gratuitous ARP on ens33 for 192.168.217.100
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: Sending gratuitous ARP on ens33 for 192.168.217.100
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: Sending gratuitous ARP on ens33 for 192.168.217.100
Oct 27 19:01:50 master1 ntpd[757]: Listen normally on 10 ens33 192.168.217.100 UDP 123

其中关于选举的部分是这样的：

脚本执行成功
由于脚本成功，优先级从98跳跃到100
强制重新选举
master1当选为master
master1的keepalived进入master状态


ct 27 19:01:46 master1 Keepalived_vrrp[1037]: VRRP_Script(check_haproxy) succeeded
Oct 27 19:01:47 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Changing effective priority from 98 to 100
Oct 27 19:01:47 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) forcing a new MASTER election
Oct 27 19:01:48 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Transition to MASTER STATE
Oct 27 19:01:49 master1 Keepalived_vrrp[1037]: VRRP_Instance(VI_1) Entering MASTER STATE

四，etcd集群的证书传递

要使用外部etcd集群需要先安装etcd集群，然后etcd的证书要分发到全部节点中，本例是四个节点，都需要分发，分发命令为：

在所有节点建立相关目录（四个服务器都执行）：

mkdir -p /etc/kubernetes/pki/etcd/

在master1节点复制etcd的证书到上述建立的目录内：


cp /opt/etcd/ssl/ca.pem /etc/kubernetes/pki/etcd/
cp /opt/etcd/ssl/server.pem  /etc/kubernetes/pki/etcd/apiserver-etcd-client.pem
cp /opt/etcd/ssl/server-key.pem  /etc/kubernetes/pki/etcd/apiserver-etcd-client-key.pem

分发etcd证书到各个节点：


scp /etc/kubernetes/pki/etcd/*  master2:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/*  master3:/etc/kubernetes/pki/etcd/
scp /etc/kubernetes/pki/etcd/*  node1:/etc/kubernetes/pki/etcd/

五，

kubeadm部署集群可以使用两种形式，第一，命令行初始化，第二，config配置文件初始化。本文选择使用config配置文件的方式：

在192.168.217.19服务器上生成模板文件；

kubeadm config print init-defaults  >kubeadm-init1-ha.yaml

编辑此模板文件，最终的内容应该如下：


[root@master1 ~]# cat kubeadm-init-ha.yaml 
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: "0"
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.217.19
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: master1
  taints: null
---
controlPlaneEndpoint: "192.168.217.100"
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  external:
    endpoints:     #下面为自定义etcd集群地址
    - https://192.168.217.19:2379
    - https://192.168.217.20:2379
    - https://192.168.217.21:2379
    caFile: /etc/kubernetes/pki/etcd/ca.pem
    certFile: /etc/kubernetes/pki/etcd/apiserver-etcd-client.pem
    keyFile: /etc/kubernetes/pki/etcd/apiserver-etcd-client-key.pem
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.22.2
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
scheduler: {}

kubeadm-init-ha.yaml配置文件说明：

配置说明：

name: 同 /etc/hosts 中统一设置的 hostname 一致，因计划在master1节点执行初始化，因此，是master1的主机名
controlPlaneEndpoint：为vip地址，可以不需要指定端口
imageRepository:由于国内无法访问google镜像仓库k8s.gcr.io，这里指定为阿里云镜像仓库registry.aliyuncs.com/google_containers
podSubnet:指定的IP地址段与后续部署的网络插件相匹配，这里需要部署flannel插件，所以配置为10.244.0.0/16

初始化命令以及该命令的输出日志（在master1节点上执行）：


[root@master1 ~]# kubeadm init --config=kubeadm-init-ha.yaml --upload-certs
[init] Using Kubernetes version: v1.22.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master1] and IPs [10.96.0.1 192.168.217.19 192.168.217.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 11.508717 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
d14a06a73a7aabf5cce805880c085f7427247d507ba01bf2d2f21c294aeb8643
[mark-control-plane] Marking the node master1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
 
Your Kubernetes control-plane has initialized successfully!
 
To start using your cluster, you need to run the following as a regular user:
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
Alternatively, if you are the root user, you can run:
 
  export KUBECONFIG=/etc/kubernetes/admin.conf
 
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
 
You can now join any number of the control-plane node running the following command on each as root:
 
  kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e \
	--control-plane --certificate-key d14a06a73a7aabf5cce805880c085f7427247d507ba01bf2d2f21c294aeb8643
 
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
 
Then you can join any number of worker nodes by running the following on each as root:
 
kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e

可以看到有两个join命令，这两个join命令是有不同用途的，

第一个join命令：

此命令是用于master节点加入的，也就是带有apiserver的节点加入，因此，在本例中，这个join命令应该用在master2和master3这两个服务器上


kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e \
	--control-plane --certificate-key d14a06a73a7aabf5cce805880c085f7427247d507ba01bf2d2f21c294aeb8643

在master2节点上执行此join命令，输入日志如下：


[root@master2 ~]# kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e --control-plane --certificate-key d14a06a73a7aabf5cce805880c085f7427247d507ba01bf2d2f21c294aeb8643
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master2] and IPs [10.96.0.1 192.168.217.20 192.168.217.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Skipping etcd check in external mode
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[control-plane-join] using external etcd - no local stacked instance added
The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node master2 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
 
This node has joined the cluster and a new control plane instance was created:
 
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
 
 
To start administering your cluster from this node, you need to run the following as a regular user:
 
	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
Run 'kubectl get nodes' to see this node join the cluster.

关键代码：

此时master2作为control-plane角色加入了集群，并且打了NoSchedule的污点。


[control-plane-join] using external etcd - no local stacked instance added
The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node master2 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

同样的在master3服务器上执行此命令，输出基本相同就不废话了。

工作节点join命令：


kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e

此命令在node1节点执行，当然，本例是只有一个工作节点，后续要在添加工作节点，仍然使用此命令即可。


kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e

此命令输出日志如下：


[root@node1 ~]# kubeadm join 192.168.217.100:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:f9ea1a25922e715f893c360759c9f56a26b1a3760df4ce140e94610a76ca6a9e 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
 
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
 
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

关键代码：

这个节点加入了集群，并且向apiserver发出了证书申请，申请被接受并返回给此节点了。


This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received

附:1：kubeadm部署完成后的环境变量问题

在上述的master初始化节点日志中，有如下输出：


 
To start using your cluster, you need to run the following as a regular user:
 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
 
Alternatively, if you are the root user, you can run:
 
  export KUBECONFIG=/etc/kubernetes/admin.conf

建议只在一个节点使用上述命令，例如，只在master1节点上做集群管理工作，那么就在master1节点上执行：


 
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果指定在工作节点做集群管理工作，需要将任意一个master节点的/etc/kubernetes/admin.conf 文件拷贝到工作节点的同名目录下，本例是node1节点：

master1节点拷贝文件：


[root@master1 ~]# scp /etc/kubernetes/admin.conf node1:/etc/kubernetes/
admin.conf

在node1节点上执行以下命令：


[root@node1 ~]#  mkdir -p $HOME/.kube
[root@node1 ~]#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@node1 ~]#   sudo chown $(id -u):$(id -g) $HOME/.kube/config

集群管理：


[root@node1 ~]# kubectl get no,po -A
NAME           STATUS   ROLES                  AGE   VERSION
node/master1   Ready    control-plane,master   22h   v1.22.2
node/master2   Ready    control-plane,master   22h   v1.22.2
node/master3   Ready    control-plane,master   21h   v1.22.2
node/node1     Ready                     21h   v1.22.2
 
NAMESPACE     NAME                                  READY   STATUS            RESTARTS       AGE
default       pod/dns-test                          0/1     Completed         0              6h11m
default       pod/nginx-7fb9867-jfg2r               1/1     Running           2              6h6m
kube-system   pod/coredns-7f6cbbb7b8-7c85v          1/1     Running           4              20h
kube-system   pod/coredns-7f6cbbb7b8-h9wtb          1/1     Running           4              20h
kube-system   pod/kube-apiserver-master1            1/1     Running           7              22h
kube-system   pod/kube-apiserver-master2            1/1     Running           6              22h
kube-system   pod/kube-apiserver-master3            0/1     Running           5              21h
kube-system   pod/kube-controller-manager-master1   1/1     Running           0              71m
kube-system   pod/kube-controller-manager-master2   1/1     Running           0              67m
kube-system   pod/kube-controller-manager-master3   0/1     Running           0              3s
kube-system   pod/kube-flannel-ds-7dmgh             1/1     Running           6              21h
kube-system   pod/kube-flannel-ds-b99h7             1/1     Running           19 (70m ago)   21h
kube-system   pod/kube-flannel-ds-hmzj7             1/1     Running           5              21h
kube-system   pod/kube-flannel-ds-vvld8             0/1     PodInitializing   4 (140m ago)   21h
kube-system   pod/kube-proxy-nkgdf                  1/1     Running           5              21h
kube-system   pod/kube-proxy-rb9zk                  1/1     Running           5              21h
kube-system   pod/kube-proxy-rvbb7                  1/1     Running           7              22h
kube-system   pod/kube-proxy-xmrp5                  1/1     Running           6              22h
kube-system   pod/kube-scheduler-master1            1/1     Running           0              71m
kube-system   pod/kube-scheduler-master2            1/1     Running           0              67m
kube-system   pod/kube-scheduler-master3            0/1     Running           0              3s

附2：kubeadm部署的证书期限问题


[root@master1 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 27, 2023 11:19 UTC   364d                                    no      
apiserver                  Oct 27, 2023 11:19 UTC   364d            ca                      no      
apiserver-kubelet-client   Oct 27, 2023 11:19 UTC   364d            ca                      no      
controller-manager.conf    Oct 27, 2023 11:19 UTC   364d                                    no      
front-proxy-client         Oct 27, 2023 11:19 UTC   364d            front-proxy-ca          no      
scheduler.conf             Oct 27, 2023 11:19 UTC   364d                                    no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 24, 2032 11:19 UTC   9y              no      
front-proxy-ca          Oct 24, 2032 11:19 UTC   9y              no

这个证书期限问题后面单独写一个博文解释如何修改。

附3：kubeadm部署的一个bug

利用kubeadm部署的集群的状态检查可以看到是不正确的：


[root@node1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Healthy     ok                                                                                            
etcd-1               Healthy     {"health":"true"}                                                                             
etcd-2               Healthy     {"health":"true"}                                                                             
etcd-0               Healthy     {"health":"true"}

在三个master节点都操作一次：


[root@master1 ~]# cd /etc/kubernetes/manifests/
[root@master1 manifests]# ll
total 12
-rw------- 1 root root 3452 Oct 27 19:19 kube-apiserver.yaml
-rw------- 1 root root 2893 Oct 27 19:19 kube-controller-manager.yaml
-rw------- 1 root root 1479 Oct 27 19:19 kube-scheduler.yaml

编辑kube-controller-manager.yaml和kube-scheduler.yaml这两个文件，将--port=0 字段删除即可，任何服务都不需要重启，即可看到集群状态正常了：

注意，此命令仅仅是快速查看集群的状态，知道集群的组件是否正常，当然也就知道有哪些组件了，可以看到我这个集群是有一个三节点的etcd集群。


[root@node1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}

相关阅读:
Windows Server 系统各版本及授权说明（附下载地址
 03 C++ 字符串、向量和数组
 C#练习题1和2
【数据结构】【程序填空】赫夫曼编码
 Java_Validation分组校验
 VB自定义版影音播放器
 简单博客网页
 基于MATLAB的数字图像处理（指纹增强）
如何使用ArcGIS Pro进行选房分析
 如何成为shopee店铺优选卖家—扬帆凌远
原文地址：https://blog.csdn.net/alwaysbefine/article/details/127528851