• kubernetes二进制安装教程单master

    前言：

     

    kubernetes的二进制部署是比较繁琐的，需要注意的细节非常多，但，二进制部署的好处也是显而易见的，能够对k8s的整体架构有更深的理解，后期的问题排查也会更加的有思路。

    k8s集群的部署是需要讲顺序的，你不能先安装一个kube-apiserver，在安装一个kubelet，然后在安装kube-controller-manage，如果这样部署，那么是不会成功的。因此，在安装前，我们需要有一个比较科学的部署规划。

    其次，二进制部署集群其实是有一些节点事件的，什么是节点事件？就是在此事情完成后，就可以进入下一个阶段了，而下一个阶段是可选择的多方向的部署。例如：

    1. [root@master cfg]# k get no -A
    2. NAME         STATUS   ROLES    AGE   VERSION
    3. k8s-master   Ready       12h   v1.18.3
    4. k8s-node1    Ready       11h   v1.18.3
    5. k8s-node2    Ready       11h   v1.18.3

     通常，安装到这的时候，就表示一个k8s集群已经算是建立好了，能用吗？可以用，但，功能是不全的，比如，coredns，这个是没有安装的，在此之前，我们也有网络插件的选择问题。然后在集群内dns安装完毕后，又进入一个新阶段：安装kubernetes的图形化管理控制界面选择，这里有N个选择，也可能选择dashboard，也可能选择kubesphere或者其它的管理界面，如果这一步完成了，又进入了下一步的安装阶段，ingress和dash-ingress的安装。其后，还有master节点的高可用部署，apiserver的高可用部署。这些都做完了，那么，才能说一个完整的可用于生产的k8s集群部署完成了。

    安装所需的相关文件：

    链接：https://pan.baidu.com/s/1XOeUD2qQYBsVQVfnulS7lA?pwd=k8ss 
    提取码：k8ss 
     

    一.集群规划

    集群规划
    序号IP角色Hostname安装组件
    1192.168.217.16master，nodek8s-maseter
    Apiserver,ControllerManager,Scheduler,Kubelet,Proxy，docker基础环境，etcd
    2192.168.217.17nodek8s-node1
    Kubelet,Proxy,Etcd，docker基础环境
    3192.168.217.18nodek8s-node2Kubelet,Proxy,Etcd，docker基础环境

    该集群的安装顺序计划为：

    1，ssh免密（全部三台服务器）

    2，时间服务器搭建（全部三台服务器）

    3，关闭swap（全部三台服务器）

    4，升级系统内核到高版本（全部三台服务器）

    5，搭建本地仓库（全部三台服务器）

    6，docker环境搭建（全部三台服务器）

    7，etcd集群搭建（全部三台服务器）

    8，kube-apiserver服务配置和安装（仅master节点）

    9，kube-controller-manager服务配置和安装（仅master节点）

    10，kube-scheduler服务配置和安装（仅master节点）

    11，验证查询集群状态---此时是第一个小阶段部署完成

    12，kubelet服务配置和安装（node节点，master节点也可安装）

    13，kube-proxy服务配置和安装（node节点，master节点也可安装）

    14，CNI网络部署---kube-flannel（全部三台服务器）

    15，集群节点状态验证查询---此时第二个小阶段部署完成。

    二，按以上步骤开始部署

    （1）三台服务器之间的ssh免密

    1. ssh-keygen -t rsa
    2. 一路回车到底，不用犹豫
    3. ssh-copy-id 192.168.217.16
    4. ssh-copy-id 192.168.217.17
    5. ssh-copy-id 192.168.217.18

    三台服务器都执行一遍，假设sshd服务没有更换端口，使用的是默认端口。

    （2）时间服务器搭建

    请看另一个博文Linux ntp时间服务器的搭建和配置_zsk_john的博客-CSDN博客_linux ntp服务器搭建

    （3）swap的关闭

    KVM虚拟机管理工作二（虚拟机磁盘优化，Centos进入dracut模式，报 /dev/centos/swap does not exist,如何恢复）_zsk_john的博客-CSDN博客_dracut模式

    这里是有误区的，如果是lvm磁盘的话，普通的swap当我没说了，反正看这个博文可以保证无缺陷的卸载swap。

    （4）升级内核

    Linux centos7升级内核（两种方法：内核编译和yum更新）_zsk_john的博客-CSDN博客_centos升级内核

    升级内核是为了集群运行更稳定，如果是低版本内核，可能会出现经常经常性集群宕机，升级到5内核以上就可以了。

    1. [root@master ~]# uname -a
    2. Linux master 5.16.9-1.el7.elrepo.x86_64 #1 SMP PREEMPT Thu Feb 10 10:39:14 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

    内核修改：

    如果内核是 4.18改为nf_conntrack_ipv4

    1. cat > /etc/modules-load.d/50-kubernetes.conf <<EOF
    2. # Load some kernel modules needed by kubernetes at boot
    3. ip_vs
    4. ip_vs_lc
    5. ip_vs_wlc
    6. ip_vs_rr
    7. ip_vs_wrr
    8. ip_vs_lblc
    9. ip_vs_lblcr
    10. ip_vs_dh
    11. ip_vs_sh
    12. ip_vs_fo
    13. ip_vs_nq
    14. ip_vs_sed
    15. ip_vs_ftp
    16. ip_vs_sh
    17. nf_conntrack 
    18. ip_tables
    19. ip_set
    20. xt_set
    21. ipt_set
    22. ipt_rpfilter
    23. ipt_REJECT
    24. ipip
    25. EOF

     

    内核优化：

    1. cat > /etc/sysctl.d/50-kubernetes.conf <<EOF
    2. net.ipv4.ip_forward=1
    3. net.bridge.bridge-nf-call-iptables=1
    4. net.bridge.bridge-nf-call-ip6tables=1
    5. fs.inotify.max_user_watches=525000
    6. fs.may_detach_mounts = 1
    7. net.ipv4.conf.all.route_localnet = 1
    8. vm.overcommit_memory=1
    9. vm.panic_on_oom=0
    10. fs.file-max=52706963
    11. fs.nr_open=52706963
    12. net.netfilter.nf_conntrack_max=2310720
    13. net.ipv4.tcp_keepalive_time = 600
    14. net.ipv4.tcp_keepalive_probes = 3
    15. net.ipv4.tcp_keepalive_intvl =15
    16. net.ipv4.tcp_max_tw_buckets = 36000
    17. net.ipv4.tcp_tw_reuse = 1
    18. net.ipv4.tcp_max_orphans = 327680
    19. net.ipv4.tcp_orphan_retries = 3
    20. net.ipv4.tcp_syncookies = 1
    21. net.ipv4.tcp_max_syn_backlog = 16384
    22. net.ipv4.ip_conntrack_max = 65536
    23. net.ipv4.tcp_max_syn_backlog = 16384
    24. net.ipv4.tcp_timestamps = 0
    25. net.core.somaxconn = 16384
    26. EOF

     以上操作所有节点均做，修改和优化后最好是重启服务器。

    （5）本地仓库的搭建

    Linux的完全本地仓库搭建指南（科普扫盲贴）_zsk_john的博客-CSDN博客_linux创建本地仓库

    本地仓库搭建是为了可能会用到的一些依赖安装。

    （6）docker环境搭建

    利用ansible的角色快速批量一键部署基础docker环境_zsk_john的博客-CSDN博客_ansible批量部署docker应用

    我写的这个博文里有ansible和docker一体化安装包，按教程搭建即可。

    （7）etcd集群搭建

    centos7操作系统 ---ansible剧本离线快速部署etcd集群_zsk_john的博客-CSDN博客_ansible离线部署

    这个也是使用ansible搭建的。

    这几步可以算作基础环境的搭建，后面的步骤将是k8s的主要核心服务搭建了。

    （8）kube-apiserver服务的搭建（master节点）
     

    准备服务运行所需要的可执行文件：

    1. mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
    2. tar zxvf kubernetes-server-linux-amd64.tar.gz
    3. cd kubernetes/server/bin
    4. cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
    5. cp kubectl /usr/bin/
    6.  
    7. chmod a+x /opt/kubernetes/bin/*
    8. chmod a+x /usr/bin/kubectl

    准备服务运行时需要的相关配置文件：

    vim /opt/kubernetes/cfg/kube-apiserver.conf 

    1. KUBE_APISERVER_OPTS="--v=2 \
    2. --logtostderr=false \
    3. --log-dir=/opt/kubernetes/logs \
    4. --etcd-servers=https://192.168.217.16:2379,https://192.168.217.17:2379,https://192.168.217.18:2379 \
    5. --bind-address=192.168.217.16 \
    6. --secure-port=6443 \
    7. --advertise-address=192.168.217.16 \
    8. --allow-privileged=true \
    9. --service-cluster-ip-range=10.0.0.0/16 \
    10. --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
    11. --authorization-mode=RBAC,Node \
    12. --enable-bootstrap-token-auth=true \
    13. --token-auth-file=/opt/kubernetes/cfg/token.csv \
    14. --service-node-port-range=30000-32767 \
    15. --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \
    16. --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
    17. --tls-cert-file=/opt/kubernetes/ssl/server.pem \
    18. --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
    19. --client-ca-file=/opt/kubernetes/ssl/ca.pem \
    20. --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
    21. --etcd-cafile=/opt/etcd/ssl/ca.pem \
    22. --etcd-certfile=/opt/etcd/ssl/server.pem \
    23. --etcd-keyfile=/opt/etcd/ssl/server-key.pem \
    24. --audit-log-maxage=30 \
    25. --audit-log-maxbackup=3 \
    26. --audit-log-maxsize=100 \
    27. --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

    kube-apiserver.conf 这个文件的配置说明：

    1. 注：上面两个\ \ 第一个是转义符，第二个是换行符，使用转义符是为了使用EOF保留换行符。
    2. --logtostderr：启用日志
    3. ---v：日志等级
    4. --log-dir：日志目录
    5. --etcd-servers：etcd集群地址
    6. --bind-address：监听地址
    7. --secure-port：https安全端口
    8. --advertise-address：集群通告地址
    9. --allow-privileged：启用授权
    10. --service-cluster-ip-range：Service虚拟IP地址段
    11. --enable-admission-plugins：准入控制模块
    12. --authorization-mode：认证授权，启用RBAC授权和节点自管理
    13. --enable-bootstrap-token-auth：启用TLS bootstrap机制
    14. --token-auth-file：bootstrap token文件
    15. --service-node-port-range：Service nodeport类型默认分配端口范围
    16. --kubelet-client-xxx：apiserver访问kubelet客户端证书
    17. --tls-xxx-file：apiserver https证书
    18. --etcd-xxxfile：连接Etcd集群证书
    19. --audit-log-xxx：审计日志

    证书文件的准备（三个文件的准备）；

    自签证书颁发机构（CA）

    vim /opt/kubernetes/ssl/ca-config.json 

    1. {
    2. "signing": {
    3. "default": {
    4. "expiry": "87600h"
    5. },
    6. "profiles": {
    7. "kubernetes": {
    8. "expiry": "87600h",
    9. "usages": [
    10. "signing",
    11. "key encipherment",
    12. "server auth",
    13. "client auth"
    14. ]
    15. }
    16. }
    17. }
    18. }

    vim /opt/kubernetes/ssl/ca-csr.json 

    1.  {
    2. "CN": "kubernetes",
    3. "key": {
    4. "algo": "rsa",
    5. "size": 2048
    6. },
    7. "names": [
    8. {
    9. "C": "CN",
    10. "L": "Beijing",
    11. "ST": "Beijing","O": "k8s",
    12. "OU": "System"
    13. }
    14. ]
    15. }

     生成证书：

    cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

    这将会生成两个证书文件，ca打头，pem后缀的证书文件。 

    使用自签CA签发kube-apiserver HTTPS证书：

    vim /opt/kubernetes/ssl/server-csr.json

    1. {
    2. "CN": "kubernetes",
    3. "hosts": [
    4. "10.0.0.1",
    5. "127.0.0.1",
    6. "192.168.217.16",
    7. "192.168.217.17",
    8. "192.168.217.18",
    9. "kubernetes",
    10. "kubernetes.default",
    11. "kubernetes.default.svc",
    12. "kubernetes.default.svc.cluster",
    13. "kubernetes.default.svc.cluster.local"
    14. ],
    15. "key": {
    16. "algo": "rsa",
    17. "size": 2048
    18. },
    19. "names": [
    20. {
    21. "C": "CN",
    22. "L": "BeiJing",
    23. "ST": "BeiJing",
    24. "O": "k8s",
    25. "OU": "System"
    26. }
    27. ]
    28. }

    生成证书：

    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

    总共会生成4个证书文件，这四个文件是以pem为后缀的，将这四个文件拷贝到 /opt/kubernetes/ssl 目录下：

    cp server*.pem ca*.pem /opt/kubernetes/ssl/

    证书生成的工作就到这告一段落了。下面是启用 TLS Bootstrapping 自签机制。

    1. cat > /opt/kubernetes/cfg/token.csv << EOF
    2. c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:nodebootstrapper"
    3. EOF

    这里的token可以使用下面的命令生成然后替换：

    head -c 16 /dev/urandom | od -An -t x | tr -d ' '

    kube-apiserver的启动脚本：

    vim /usr/lib/systemd/system/kube-apiserver.service 

    1. [Unit]
    2. Description=Kubernetes API Server
    3. Documentation=https://github.com/kubernetes/kubernetes
    4. [Service]
    5. EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
    6. ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
    7. Restart=on-failure
    8. [Install]
    9. WantedBy=multi-user.target

    服务启动和加入自启：

    1. systemctl daemon-reload
    2. systemctl start kube-apiserver
    3. systemctl enable kube-apiserver

    该服务状态为绿色表示正常：

    1. [root@master ssl]# systemctl status kube-apiserver
    2. ● kube-apiserver.service - Kubernetes API Server
    3.    Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
    4.    Active: active (running) since Fri 2022-08-26 15:33:19 CST; 6h ago
    5.      Docs: https://github.com/kubernetes/kubernetes
    6.  Main PID: 3009 (kube-apiserver)
    7.    Memory: 365.3M
    8.    CGroup: /system.slice/kube-apiserver.service
    9.            └─3009 /opt/kubernetes/bin/kube-apiserver --v=2 --logtostderr=false --log-dir=/opt/kubernetes/logs --etcd-servers=https://192.168.217.16:2379,https://1...
    10.  
    11. Aug 26 15:33:19 master systemd[1]: Started Kubernetes API Server.
    12. Aug 26 15:33:19 master systemd[1]: Starting Kubernetes API Server...
    13. Aug 26 15:33:28 master kube-apiserver[3009]: E0826 15:33:28.034854    3009 controller.go:152] Unable to remove old endpoints from kubernetes service: Sto...ErrorMsg:
    14. Hint: Some lines were ellipsized, use -l to show in full.

    如果有错误导致服务未能正常启动，可查看系统日志 /var/log/messages

    通过对/var/log/messages日志的观察，可以发现，在第一次启动apiserver的时候，生成了非常多的角色，这些角色对应了k8s内的各种资源。例如：

    1. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.321342    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/cluster-admin
    2. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.335178    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:discovery
    3. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.346905    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:basic-user
    4. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.359675    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:public-info-viewer
    5. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.370449    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/admin
    6. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.381805    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/edit
    7. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.395624    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/view
    8. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.406568    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:aggregate-to-admin
    9. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.415029    6822 healthz.go:200] [+]ping ok
    1. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.516294    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-aggregator
    2. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.525808    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-controller-manager
    3. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.535778    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-dns
    4. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.545944    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:persistent-volume-provisioner
    5. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.558356    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:nodeclient
    6. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.567806    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
    7. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.577033    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:volume-scheduler
    8. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.585929    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:legacy-unknown-approver
    9. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.596499    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:kubelet-serving-approver
    10. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.605861    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:ku
    1. g 30 10:01:23 master kube-apiserver: I0830 10:01:23.614996    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:kube-apiserver-client-kubelet-approver
    2. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.624625    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:node-proxier
    3. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.635380    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:kube-scheduler
    4. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.644132    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:attachdetach-controller
    5. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.653821    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:clusterrole-aggregation-controller
    6. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.663108    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:cronjob-controller
    7. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.672682    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:daemon-set-controller
    8. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.685326    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:deployment-controller
    9. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.694401    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:disruption-controller
    10. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.703354    6822 storage_rbac.go:220] created clusterrole.rbac.authorization.k8s.io/system:controller:endpoint-controller
    11. Aug 30 10:01:23 master kube-apiserver: I0830 10:01:23.713226    6822 healthz.go:200] [+]ping ok
    12. Aug 30 10:01:23 master kube-apiserver: [+]log ok
    13. Aug 30 10:01:23 master kube-apiserver: [+]etcd ok

    1. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.123145    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:endpointslice-controller
    2. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.132424    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:expand-controller
    3. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.149014    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:generic-garbage-collector
    4. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.160210    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:horizontal-pod-autoscaler
    5. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.169018    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:job-controller
    6. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.178514    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:namespace-controller
    7. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.187484    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:node-controller
    8. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.201137    6822 storage_rbac.go:248] created clusterrolebinding.rbac.authorization.k8s.io/system:controller:persistent-volume-binder
    9. Aug 30 10:01:24 master kube-apiserver: I0830 10:01:24.213896    6822 healthz.go:200] [+]ping ok

    （9）部署kube-controller-manager

    该服务的配置文件：

    vim /opt/kubernetes/cfg/kube-controller-manager.conf

    1.  KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
    2. --v=2 \
    3. --log-dir=/opt/kubernetes/logs \
    4. --leader-elect=true \
    5. --master=127.0.0.1:8080 \
    6. --bind-address=127.0.0.1 \
    7. --allocate-node-cidrs=true \
    8. --cluster-cidr=10.244.0.0/16 \
    9. --service-cluster-ip-range=10.0.0.0/16 \
    10. --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
    11. --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
    12. --root-ca-file=/opt/kubernetes/ssl/ca.pem \
    13. --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
    14. --experimental-cluster-signing-duration=87600h0m0s"

    配置文件说明：

    1. --master：通过本地非安全本地端口8080连接apiserver。
    2. --leader-elect：当该组件启动多个时，自动选举（HA）
    3. --cluster-signing-cert-file/--cluster-signing-key-file：自动为kubelet颁发证书的CA，与apiserver保持一致，也就是两个服务共用ca证书。

    该服务的启动脚本：

    vim /usr/lib/systemd/system/kube-controller-manager.service 

    1. [Unit]
    2. Description=Kubernetes Controller Manager
    3. Documentation=https://github.com/kubernetes/kubernetes
    4. [Service]
    5. EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
    6. ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
    7. Restart=on-failure
    8. [Install]
    9. WantedBy=multi-user.target

    启动并设置开机启动：

    1. systemctl daemon-reload
    2. systemctl start kube-controller-manager
    3. systemctl enable kube-controller-manager

    （10）部署kube-scheduler

    这个服务是调度服务，主要调度各类资源的，通过和controller-manage服务通信，以及etcd通知进行各类资源调度。

    配置文件：

    vim /opt/kubernetes/cfg/kube-scheduler.conf

    1. KUBE_SCHEDULER_OPTS="--logtostderr=false \
    2. --v=2 \
    3. --log-dir=/opt/kubernetes/logs \
    4. --leader-elect \
    5. --master=127.0.0.1:8080 \
    6. --bind-address=127.0.0.1"

    配置文件说明：

    1. --master：通过本地非安全本地端口8080连接apiserver。
    2. --leader-elect：当该组件启动多个时，自动选举（HA）

    启动脚本：

    vim /usr/lib/systemd/system/kube-scheduler.service

    1. [Unit]
    2. Description=Kubernetes Scheduler
    3. Documentation=https://github.com/kubernetes/kubernetes
    4. [Service]
    5. EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
    6. ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
    7. Restart=on-failure
    8. [Install]
    9. WantedBy=multi-user.target

    启动并设置开机启动：

    1. systemctl daemon-reload
    2. systemctl start kube-scheduler
    3. systemctl enable kube-scheduler

    （11）

    此时，这三个服务搭建完毕后，就可以集群的健康检查了：

    1. [root@master cfg]# kubectl get cs
    2. NAME                 STATUS    MESSAGE             ERROR
    3. scheduler            Healthy   ok                  
    4. controller-manager   Healthy   ok                  
    5. etcd-0               Healthy   {"health":"true"}   
    6. etcd-1               Healthy   {"health":"true"}   
    7. etcd-2               Healthy   {"health":"true"}

    如果哪个服务没有启动或者异常，此命令都会显示出来。例如，停止一个etcd，上面的命令将会报告错误：

    1. [root@master cfg]# kubectl get cs
    2. NAME                 STATUS      MESSAGE                                                                                             ERROR
    3. scheduler            Healthy     ok                                                                                                  
    4. etcd-1               Unhealthy   Get https://192.168.217.17:2379/health: dial tcp 192.168.217.17:2379: connect: connection refused   
    5. controller-manager   Healthy     ok                                                                                                  
    6. etcd-0               Healthy     {"health":"true"}                                                                                   
    7. etcd-2               Healthy     {"health":"true"}

    （12）node节点安装kubelet

    kubelet服务是node工作节点比较重要的一个服务，这个服务也不太好配置：

    kubelet服务的配置文件：

    vim /opt/kubernetes/cfg/kubelet.conf 

    1. KUBELET_OPTS="--logtostderr=false \
    2. --v=2 \
    3. --log-dir=/opt/kubernetes/logs \
    4. --hostname-override=k8s-master \
    5. --network-plugin=cni \
    6. --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
    7. --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
    8. --config=/opt/kubernetes/cfg/kubelet-config.yml \
    9. --cert-dir=/opt/kubernetes/ssl \
    10. --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"

    导入相关镜像包，包名是registry.cn-hangzhou.aliyuncs.com_google_containers_pause_3.2.tar，三个节点都导入。

    配置文件说明：

    1. --hostname-override：显示名称，集群中唯一
    2. --network-plugin：启用CNI
    3. --kubeconfig：空路径，会自动生成，后面用于连接apiserver
    4. --bootstrap-kubeconfig：首次启动向apiserver申请证书
    5. --config：配置参数文件
    6. --cert-dir：kubelet证书生成目录
    7. --pod-infra-container-image：管理Pod网络容器的镜像

    这里要注意一个难点，-kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig这一段，表示这个文件会在服务启动的时候自动生成，但一般稍微有点错，它就生成不了，比如下面的文件如果有写错，那么，将不会自动生成这个文件。

    vim /opt/kubernetes/cfg/kubelet-config.yml

    1. kind: KubeletConfiguration
    2. apiVersion: kubelet.config.k8s.io/v1beta1
    3. address: 0.0.0.0
    4. port: 10250
    5. readOnlyPort: 10255
    6. cgroupDriver: cgroupfs
    7. clusterDNS:
    8.   - 10.0.0.2
    9. clusterDomain: cluster.local 
    10. failSwapOn: false
    11. authentication:
    12.   anonymous:
    13.     enabled: false
    14.   webhook:
    15.     cacheTTL: 2m0s
    16.     enabled: true
    17.   x509:
    18.     clientCAFile: /opt/kubernetes/ssl/ca.pem 
    19. authorization:
    20.   mode: Webhook
    21.   webhook:
    22.     cacheAuthorizedTTL: 5m0s
    23.     cacheUnauthorizedTTL: 30s
    24. evictionHard:
    25.   imagefs.available: 15%
    26.   memory.available: 100Mi
    27.   nodefs.available: 10%
    28.   nodefs.inodesFree: 5%
    29. maxOpenFiles: 1000000
    30. maxPods: 110

    生成bootstrap.kubeconfig文件：

    1. KUBE_APISERVER="https://192.168.217.16:6443"
    2. TOKEN="c47ffb939f5ca36231d9e3121a252940"

    集群名称的定义在下面这个文件内，这里定义的名称是kubernetes： 

    1. kubectl config set-cluster kubernetes \
    2. --certificate-authority=/opt/kubernetes/ssl/ca.pem \
    3. --embed-certs=true \
    4. --server=${KUBE_APISERVER} \
    5. --kubeconfig=bootstrap.kubeconfig

    定义的用户名称是kubelet-bootstrap，这个用户需要授予admin权限。 

    1. kubectl config set-credentials "kubelet-bootstrap" \
    2. --token=${TOKEN} \
    3. --kubeconfig=bootstrap.kubeconfig

    生成config文件，这个文件非常重要，在执行命令的当前目录生成，如果不是在/opt/kubernetes/cfs目录下执行的此命令，需要copy这个文件到前述目录内。 

    1. kubectl config set-context default \
    2. --cluster=kubernetes \
    3. --user="kubelet-bootstrap" \
    4. --kubeconfig=bootstrap.kubeconfig
    kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

    权限授予：

    1. kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
    2. kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=cluster-admin  --user=kubelet-bootstrap

    授权apiserver访问kubelet ：

    vim apiserver-to-kubelet-rbac.yaml 

    1. apiVersion: rbac.authorization.k8s.io/v1
    2. kind: ClusterRole
    3. metadata:
    4.   annotations:
    5.     rbac.authorization.kubernetes.io/autoupdate: "true"
    6.   labels:
    7.     kubernetes.io/bootstrapping: rbac-defaults
    8.   name: system:kube-apiserver-to-kubelet
    9. rules:
    10.   - apiGroups:
    11.     - ""
    12.     resources:
    13.       - nodes/proxy
    14.       - nodes/stats
    15.       - nodes/log
    16.       - nodes/spec
    17.       - nodes/metrics
    18.       - pods/log
    19.     verbs:
    20.       - "*"
    21. ---
    22. apiVersion: rbac.authorization.k8s.io/v1
    23. kind: ClusterRoleBinding
    24. metadata:
    25.   name: system:kube-apiserver
    26.   namespace: ""
    27. roleRef:
    28.   apiGroup: rbac.authorization.k8s.io
    29.   kind: ClusterRole
    30.   name: system:kube-apiserver-to-kubelet
    31. subjects:
    32.   - apiGroup: rbac.authorization.k8s.io
    33.     kind: User
    34.     name: kubernetes

    执行这个文件：

    kubectl apply -f apiserver-to-kubelet-rbac.yaml

    拷贝文件到配置文件存放目录：

    cp bootstrap.kubeconfig /opt/kubernetes/cfg

    kubelet服务的启动脚本：

    vim /usr/lib/systemd/system/kubelet.service 

    1. [Unit]
    2. Description=Kubernetes Kubelet
    3. After=docker.service
    4. [Service]
    5. EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
    6. ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
    7. Restart=on-failure
    8. LimitNOFILE=65536
    9. [Install]
    10. WantedBy=multi-user.target

    删除一下自动生成的文件： 

    rm -rf /usr/lib/systemd/system/kubelet.service.d

    启动并设置开机启动：

    1. systemctl daemon-reload
    2. systemctl start kubelet
    3. systemctl enable kubelet

    批准kubelet证书申请并加入集群：

    1. # 查看kubelet证书请求
    2. kubectl get csr
    3. NAME AGE SIGNERNAME
    4. REQUESTOR CONDITION
    5. node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A 6m3s kubernetes.io/kubeapiserver-client-kubelet kubelet-bootstrap Pending
    6. # 批准申请
    7. kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A

    此时在kubectl get csr 状态将变成 approve，issued表示申请通过。

    1. [root@master cfg]# k get csr
    2. NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
    3. node-csr-u3XGZBQ_M8SKt60J5jCIH7enAbRtKRsbW8LgBM8XsRQ   24m   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued

    此时查看node节点，可以看到一个notready的节点：

    1. [root@master cfg]# k get no
    2. NAME         STATUS     ROLES    AGE   VERSION
    3. k8s-master   NotReady      63s   v1.18.3

    （13）

    部署kube-proxy

    服务配置文件：

    vim /opt/kubernetes/cfg/kube-proxy.conf

    1.  KUBE_PROXY_OPTS="--logtostderr=false \
    2. --v=2 \
    3. --log-dir=/opt/kubernetes/logs \
    4. --config=/opt/kubernetes/cfg/kube-proxy-config.yml"

    配置参数文件：

    vim /opt/kubernetes/cfg/kube-proxy-config.yml 

    1. kind: KubeProxyConfiguration
    2. apiVersion: kubeproxy.config.k8s.io/v1alpha1
    3. bindAddress: 0.0.0.0
    4. metricsBindAddress: 0.0.0.0:10249
    5. clientConnection:
    6.   kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
    7. hostnameOverride: k8s-master
    8. clusterCIDR: 10.244.0.0/16

    生成kube-proxy.kubeconfig文件

    vim /opt/kubernetes/ssl/kube-proxy-csr.json 
     

    1. {"CN": "system:kube-proxy",
    2. "hosts": [],
    3. "key": {
    4. "algo": "rsa",
    5. "size": 2048
    6. },
    7. "names": [
    8. {
    9. "C": "CN",
    10. "L": "BeiJing",
    11. "ST": "BeiJing",
    12. "O": "k8s",
    13. "OU": "System"
    14. }
    15. ]
    16. }

    生成证书：

    cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes /opt/kubernetes/ssl/kube-proxy-csr.json | cfssljson -bare kube-proxy

    复制证书：

    cp kube-proxy-key.pem kube-proxy.pem /opt/kubernetes/ssl/

    生成kubeconfig文件：

    KUBE_APISERVER="https://192.168.217.16:6443"
    1. kubectl config set-cluster kubernetes \
    2. --certificate-authority=/opt/kubernetes/ssl/ca.pem \
    3. --embed-certs=true \
    4. --server=${KUBE_APISERVER} \
    5. --kubeconfig=kube-proxy.kubeconfig
    kubectl config set-credentials kube-proxy --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
    1. kubectl config set-context default \
    2. --cluster=kubernetes \
    3. --user=kube-proxy \
    4. --kubeconfig=kube-proxy.kubeconfig

    kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

    服务启动脚本：

    1. [root@master bin]# cat /usr/lib/systemd/system/kube-proxy.service 
    2. [Unit]
    3. Description=Kubernetes Proxy
    4. After=network.target
    5. [Service]
    6. EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
    7. ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
    8. Restart=on-failure
    9. LimitNOFILE=65536
    10. [Install]
    11. WantedBy=multi-user.target

    启动并设置开机启动：

    1. systemctl daemon-reload
    2. systemctl start kube-proxy
    3. systemctl enable kube-proxy

    （14）

    工作节点部署（在master节点的基础上部署）

    拷贝相关文件到node1节点（配置文件，证书文件和可执行文件以及服务启动脚本，这里是演示的node1节点，node2节点一样的操作啦）：

    先在node1服务器上建立相关文件夹：

    mkdir -p /opt/kubernetes/{cfg,bin,ssl,logs}
    1. scp /opt/kubernetes/bin/{kubelet,kube-proxy}   k8s-node1:/opt/kubernetes/bin/
    2.  
    3. scp /usr/lib/systemd/system/{kubelet.service,kube-proxy.service}   k8s-node1:/usr/lib/systemd/system/
    4.  
    5. scp /opt/kubernetes/cfg/{kubelet.conf,kube-proxy.conf,kube-proxy-config.yml,kubelet-config.yml,bootstrap.kubeconfig,kube-proxy.kubeconfig}  k8s-node1:/opt/kubernetes/cfg/
    6.  
    7. scp /opt/kubernetes/ssl/{ca-key.pem,ca.pem,kubelet.crt,kubelet.key,kube-proxy-key.pem,kube-proxy.pem,server-key.pem,server.pem}  k8s-node1:/opt/kubernetes/ssl/
    8.

    两个文件修改修改主机名：

    kube-proxy-config.yml和kubelet.conf 这两个文件里的--hostname-override=的值修改为当前主机名，比如，是在node1服务器上修改，就写k8s-node1

    剩下的都不需要改动，启动服务就好了，服务状态检查，看成功后，在master服务器上批准节点加入就可以啦（例如，node1节点的批准）：

    1. [root@master ssl]# k get csr
    2. NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
    3. node-csr-6peyUWAChHCuvf5bO75sb0SRB5xVxlnMpH1F1UKbc2U   51s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending
    4. [root@master ssl]# kubectl certificate approve node-csr-6peyUWAChHCuvf5bO75sb0SRB5xVxlnMpH1F1UKbc2U
    5. certificatesigningrequest.certificates.k8s.io/node-csr-6peyUWAChHCuvf5bO75sb0SRB5xVxlnMpH1F1UKbc2U approved

    在master上查看结果，验证是否正常：

    1. [root@master cfg]# k get no
    2. NAME         STATUS     ROLES    AGE     VERSION
    3. k8s-master   NotReady      4h40m   v1.18.3
    4. k8s-node1    NotReady      34m     v1.18.3
    5. k8s-node2    NotReady      33m     v1.18.3


    此时节点状态是没有准备，原因如下（cni插件配置没有初始化）：

    Aug 27 15:58:56 master kubelet: E0827 15:58:56.093236   14623 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

    15，

    安装网络插件

    导入docker镜像包quay.io_coreos_flannel_v0.13.0.tar，所有节点都导入，也就是docker load  

    1. [root@master cfg]# cat ../bin/kube-flannel.yml 
    2. ---
    3. apiVersion: policy/v1beta1
    4. kind: PodSecurityPolicy
    5. metadata:
    6.   name: psp.flannel.unprivileged
    7.   annotations:
    8.     seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    9.     seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    10.     apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    11.     apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    12. spec:
    13.   privileged: false
    14.   volumes:
    15.   - configMap
    16.   - secret
    17.   - emptyDir
    18.   - hostPath
    19.   allowedHostPaths:
    20.   - pathPrefix: "/etc/cni/net.d"
    21.   - pathPrefix: "/etc/kube-flannel"
    22.   - pathPrefix: "/run/flannel"
    23.   readOnlyRootFilesystem: false
    24.   # Users and groups
    25.   runAsUser:
    26.     rule: RunAsAny
    27.   supplementalGroups:
    28.     rule: RunAsAny
    29.   fsGroup:
    30.     rule: RunAsAny
    31.   # Privilege Escalation
    32.   allowPrivilegeEscalation: false
    33.   defaultAllowPrivilegeEscalation: false
    34.   # Capabilities
    35.   allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
    36.   defaultAddCapabilities: []
    37.   requiredDropCapabilities: []
    38.   # Host namespaces
    39.   hostPID: false
    40.   hostIPC: false
    41.   hostNetwork: true
    42.   hostPorts:
    43.   - min: 0
    44.     max: 65535
    45.   # SELinux
    46.   seLinux:
    47.     # SELinux is unused in CaaSP
    48.     rule: 'RunAsAny'
    49. ---
    50. kind: ClusterRole
    51. apiVersion: rbac.authorization.k8s.io/v1
    52. metadata:
    53.   name: flannel
    54. rules:
    55. - apiGroups: ['extensions']
    56.   resources: ['podsecuritypolicies']
    57.   verbs: ['use']
    58.   resourceNames: ['psp.flannel.unprivileged']
    59. - apiGroups:
    60.   - ""
    61.   resources:
    62.   - pods
    63.   verbs:
    64.   - get
    65. - apiGroups:
    66.   - ""
    67.   resources:
    68.   - nodes
    69.   verbs:
    70.   - list
    71.   - watch
    72. - apiGroups:
    73.   - ""
    74.   resources:
    75.   - nodes/status
    76.   verbs:
    77.   - patch
    78. ---
    79. kind: ClusterRoleBinding
    80. apiVersion: rbac.authorization.k8s.io/v1
    81. metadata:
    82.   name: flannel
    83. roleRef:
    84.   apiGroup: rbac.authorization.k8s.io
    85.   kind: ClusterRole
    86.   name: flannel
    87. subjects:
    88. - kind: ServiceAccount
    89.   name: flannel
    90.   namespace: kube-system
    91. ---
    92. apiVersion: v1
    93. kind: ServiceAccount
    94. metadata:
    95.   name: flannel
    96.   namespace: kube-system
    97. ---
    98. kind: ConfigMap
    99. apiVersion: v1
    100. metadata:
    101.   name: kube-flannel-cfg
    102.   namespace: kube-system
    103.   labels:
    104.     tier: node
    105.     app: flannel
    106. data:
    107.   cni-conf.json: |
    108.     {
    109.       "name": "cbr0",
    110.       "cniVersion": "0.3.1",
    111.       "plugins": [
    112.         {
    113.           "type": "flannel",
    114.           "delegate": {
    115.             "hairpinMode": true,
    116.             "isDefaultGateway": true
    117.           }
    118.         },
    119.         {
    120.           "type": "portmap",
    121.           "capabilities": {
    122.             "portMappings": true
    123.           }
    124.         }
    125.       ]
    126.     }
    127.   net-conf.json: |
    128.     {
    129.       "Network": "10.244.0.0/16",
    130.       "Backend": {
    131.         "Type": "vxlan"
    132.       }
    133.     }
    134. ---
    135. apiVersion: apps/v1
    136. kind: DaemonSet
    137. metadata:
    138.   name: kube-flannel-ds
    139.   namespace: kube-system
    140.   labels:
    141.     tier: node
    142.     app: flannel
    143. spec:
    144.   selector:
    145.     matchLabels:
    146.       app: flannel
    147.   template:
    148.     metadata:
    149.       labels:
    150.         tier: node
    151.         app: flannel
    152.     spec:
    153.       affinity:
    154.         nodeAffinity:
    155.           requiredDuringSchedulingIgnoredDuringExecution:
    156.             nodeSelectorTerms:
    157.             - matchExpressions:
    158.               - key: kubernetes.io/os
    159.                 operator: In
    160.                 values:
    161.                 - linux
    162.       hostNetwork: true
    163.       priorityClassName: system-node-critical
    164.       tolerations:
    165.       - operator: Exists
    166.         effect: NoSchedule
    167.       serviceAccountName: flannel
    168.       initContainers:
    169.       - name: install-cni
    170.         image: quay.io/coreos/flannel:v0.13.0
    171.         command:
    172.         - cp
    173.         args:
    174.         - -f
    175.         - /etc/kube-flannel/cni-conf.json
    176.         - /etc/cni/net.d/10-flannel.conflist
    177.         volumeMounts:
    178.         - name: cni
    179.           mountPath: /etc/cni/net.d
    180.         - name: flannel-cfg
    181.           mountPath: /etc/kube-flannel/
    182.       containers:
    183.       - name: kube-flannel
    184.         image: quay.io/coreos/flannel:v0.13.0
    185.         command:
    186.         - /opt/bin/flanneld
    187.         args:
    188.         - --ip-masq
    189.         - --kube-subnet-mgr
    190.         resources:
    191.           requests:
    192.             cpu: "100m"
    193.             memory: "50Mi"
    194.           limits:
    195.             cpu: "100m"
    196.             memory: "50Mi"
    197.         securityContext:
    198.           privileged: false
    199.           capabilities:
    200.             add: ["NET_ADMIN", "NET_RAW"]
    201.         env:
    202.         - name: POD_NAME
    203.           valueFrom:
    204.             fieldRef:
    205.               fieldPath: metadata.name
    206.         - name: POD_NAMESPACE
    207.           valueFrom:
    208.             fieldRef:
    209.               fieldPath: metadata.namespace
    210.         volumeMounts:
    211.         - name: run
    212.           mountPath: /run/flannel
    213.         - name: flannel-cfg
    214.           mountPath: /etc/kube-flannel/
    215.       volumes:
    216.       - name: run
    217.         hostPath:
    218.           path: /run/flannel
    219.       - name: cni
    220.         hostPath:
    221.           path: /etc/cni/net.d
    222.       - name: flannel-cfg
    223.         configMap:
    224.           name: kube-flannel-cfg

    kubectl apply -f kube-flannel.yaml  执行上面的文件就安装好了网络插件。

    （15）COREDNS的安装

    总共5个配置文件，其中一个是测试dns的文件：

    ServiceAccount的

    1. cat << EOF > coredns-ns.yaml
    2. apiVersion: v1
    3. kind: ServiceAccount
    4. metadata:
    5.   name: coredns
    6.   namespace: kube-system
    7. EOF

    coredns 的configMap： 

    1. cat << EOF > coredns-cm.yaml
    2. apiVersion: v1
    3. kind: ConfigMap
    4. metadata:
    5.   name: coredns
    6.   namespace: kube-system
    7. data:
    8.   Corefile: |
    9.     .:53 {
    10.         errors
    11.         health {
    12.           lameduck 5s
    13.         }
    14.         ready
    15.         kubernetes cluster.local in-addr.arpa ip6.arpa {
    16.           fallthrough in-addr.arpa ip6.arpa
    17.         }
    18.         prometheus :9153
    19.         forward . /etc/resolv.conf {
    20.           max_concurrent 1000
    21.         }
    22.         cache 30
    23.         loop
    24.         reload
    25.         loadbalance
    26.     }
    27. EOF

    coredns的用户角色权限清单文件： 

    1. cat << EOF > coredns-rbac.yaml
    2. apiVersion: rbac.authorization.k8s.io/v1
    3. kind: ClusterRole
    4. metadata:
    5.   labels:
    6.     kubernetes.io/bootstrapping: rbac-defaults
    7.   name: system:coredns
    8. rules:
    9.   - apiGroups:
    10.     - ""
    11.     resources:
    12.     - endpoints
    13.     - services
    14.     - pods
    15.     - namespaces
    16.     verbs:
    17.     - list
    18.     - watch
    19.   - apiGroups:
    20.     - discovery.k8s.io
    21.     resources:
    22.     - endpointslices
    23.     verbs:
    24.     - list
    25.     - watch
    26. ---
    27. apiVersion: rbac.authorization.k8s.io/v1
    28. kind: ClusterRoleBinding
    29. metadata:
    30.   annotations:
    31.     rbac.authorization.kubernetes.io/autoupdate: "true"
    32.   labels:
    33.     kubernetes.io/bootstrapping: rbac-defaults
    34.   name: system:coredns
    35. roleRef:
    36.   apiGroup: rbac.authorization.k8s.io
    37.   kind: ClusterRole
    38.   name: system:coredns
    39. subjects:
    40. - kind: ServiceAccount
    41.   name: coredns
    42.   namespace: kube-system
    43. EOF

     coredns部署文件

    1. cat << EOF > coredns-deploy.yaml
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: coredns
    6.   namespace: kube-system
    7.   labels:
    8.     k8s-app: kube-dns
    9.     kubernetes.io/name: "CoreDNS"
    10. spec:
    11.   # replicas: not specified here:
    12.   # 1. Default is 1.
    13.   # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
    14.   strategy:
    15.     type: RollingUpdate
    16.     rollingUpdate:
    17.       maxUnavailable: 1
    18.   selector:
    19.     matchLabels:
    20.       k8s-app: kube-dns
    21.   template:
    22.     metadata:
    23.       labels:
    24.         k8s-app: kube-dns
    25.     spec:
    26.       priorityClassName: system-cluster-critical
    27.       serviceAccountName: coredns
    28.       tolerations:
    29.         - key: "CriticalAddonsOnly"
    30.           operator: "Exists"
    31.       nodeSelector:
    32.         kubernetes.io/os: linux
    33.       affinity:
    34.          podAntiAffinity:
    35.            requiredDuringSchedulingIgnoredDuringExecution:
    36.            - labelSelector:
    37.                matchExpressions:
    38.                - key: k8s-app
    39.                  operator: In
    40.                  values: ["kube-dns"]
    41.              topologyKey: kubernetes.io/hostname
    42.       containers:
    43.       - name: coredns
    44.         image: registry.aliyuncs.com/google_containers/coredns:1.8.6
    45.         imagePullPolicy: IfNotPresent
    46.         resources:
    47.           limits:
    48.             memory: 170Mi
    49.           requests:
    50.             cpu: 100m
    51.             memory: 70Mi
    52.         args: [ "-conf", "/etc/coredns/Corefile" ]
    53.         volumeMounts:
    54.         - name: config-volume
    55.           mountPath: /etc/coredns
    56.           readOnly: true
    57.         ports:
    58.         - containerPort: 53
    59.           name: dns
    60.           protocol: UDP
    61.         - containerPort: 53
    62.           name: dns-tcp
    63.           protocol: TCP
    64.         - containerPort: 9153
    65.           name: metrics
    66.           protocol: TCP
    67.         securityContext:
    68.           allowPrivilegeEscalation: false
    69.           capabilities:
    70.             add:
    71.             - NET_BIND_SERVICE
    72.             drop:
    73.             - all
    74.           readOnlyRootFilesystem: true
    75.         livenessProbe:
    76.           httpGet:
    77.             path: /health
    78.             port: 8080
    79.             scheme: HTTP
    80.           initialDelaySeconds: 60
    81.           timeoutSeconds: 5
    82.           successThreshold: 1
    83.           failureThreshold: 5
    84.         readinessProbe:
    85.           httpGet:
    86.             path: /ready
    87.             port: 8181
    88.             scheme: HTTP
    89.       dnsPolicy: Default
    90.       volumes:
    91.         - name: config-volume
    92.           configMap:
    93.             name: coredns
    94.             items:
    95.             - key: Corefile
    96.               path: Corefile
    97. EOF

    coredns的service文件，这里需要注意，clusterIP的值是和kubelet-config.yml 里的值一致的。 

    1. cat << EOF > coredns-svc.yaml
    2. apiVersion: v1
    3. kind: Service
    4. metadata:
    5.   name: kube-dns
    6.   namespace: kube-system
    7.   annotations:
    8.     prometheus.io/port: "9153"
    9.     prometheus.io/scrape: "true"
    10.   labels:
    11.     k8s-app: kube-dns
    12.     kubernetes.io/cluster-service: "true"
    13.     kubernetes.io/name: "CoreDNS"
    14. spec:
    15.   selector:
    16.     k8s-app: kube-dns
    17.   clusterIP: 10.0.0.2
    18.   ports:
    19.   - name: dns
    20.     port: 53
    21.     protocol: UDP
    22.   - name: dns-tcp
    23.     port: 53
    24.     protocol: TCP
    25.   - name: metrics
    26.     port: 9153
    27.     protocol: TCP
    28. EOF

    coredns的测试：

    启动一个临时pod，以运行测试命令：

    kubectl run -i --tty --image busybox:1.28.3 -n web  dns-test --restart=Never --rm

    解析集群内外域名都没有问题，输出是以下的表示coredns功能正常。 

    1. / # nslookup kubernetes
    2. Server:    10.0.0.2
    3. Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
    4.  
    5. Name:      kubernetes
    6. Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
    7. / # nslookup kubernetes.default
    8. Server:    10.0.0.2
    9. Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
    10.  
    11. Name:      kubernetes.default
    12. Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
    13. / # nslookup baidu.com
    14. Server:    10.0.0.2
    15. Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
    16.  
    17. Name:      baidu.com
    18. Address 1: 110.242.68.66
    19. Address 2: 39.156.66.10
    20. / # cat /etc/resolv.conf 
    21. nameserver 10.0.0.2
    22. search web.svc.cluster.local svc.cluster.local cluster.local localdomain default.svc.cluster.local
    23. options ndots:5
    24.

    （16）

    dashboard的安装

    kubernetesui_metrics-scraper_v1.0.6.tar和dashboard.tar这两个网盘内的文件三节点都使用docker load 导入，因为是deployment方式部署，不清楚到底是在哪个节点部署。

    集群角色绑定

    kubectl create clusterrolebinding default --clusterrole=cluster-admin --serviceaccount=kube-system:default --namespace=kube-system

    安装部署yaml文件内容如下（文件内容比较多，执行这个文件  kubectl apply -f dashboard.yml）；

    1. [root@master ~]# cat dashboard.yml 
    2. # Copyright 2017 The Kubernetes Authors.
    3. #
    4. # Licensed under the Apache License, Version 2.0 (the "License");
    5. # you may not use this file except in compliance with the License.
    6. # You may obtain a copy of the License at
    7. #
    8. #     http://www.apache.org/licenses/LICENSE-2.0
    9. #
    10. # Unless required by applicable law or agreed to in writing, software
    11. # distributed under the License is distributed on an "AS IS" BASIS,
    12. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    13. # See the License for the specific language governing permissions and
    14. # limitations under the License.
    15.  
    16. apiVersion: v1
    17. kind: Namespace
    18. metadata:
    19.   name: kubernetes-dashboard
    20.  
    21. ---
    22.  
    23. apiVersion: v1
    24. kind: ServiceAccount
    25. metadata:
    26.   labels:
    27.     k8s-app: kubernetes-dashboard
    28.   name: kubernetes-dashboard
    29.   namespace: kubernetes-dashboard
    30.  
    31. ---
    32.  
    33. kind: Service
    34. apiVersion: v1
    35. metadata:
    36.   labels:
    37.     k8s-app: kubernetes-dashboard
    38.   name: kubernetes-dashboard
    39.   namespace: kubernetes-dashboard
    40. spec:
    41.   ports:
    42.     - port: 443
    43.       targetPort: 8443
    44.       nodePort: 30008
    45.   type: NodePort
    46.   selector:
    47.     k8s-app: kubernetes-dashboard
    48.  
    49. ---
    50.  
    51. apiVersion: v1
    52. kind: Secret
    53. metadata:
    54.   labels:
    55.     k8s-app: kubernetes-dashboard
    56.   name: kubernetes-dashboard-certs
    57.   namespace: kubernetes-dashboard
    58. type: Opaque
    59.  
    60. ---
    61.  
    62. apiVersion: v1
    63. kind: Secret
    64. metadata:
    65.   labels:
    66.     k8s-app: kubernetes-dashboard
    67.   name: kubernetes-dashboard-csrf
    68.   namespace: kubernetes-dashboard
    69. type: Opaque
    70. data:
    71.   csrf: ""
    72.  
    73. ---
    74.  
    75. apiVersion: v1
    76. kind: Secret
    77. metadata:
    78.   labels:
    79.     k8s-app: kubernetes-dashboard
    80.   name: kubernetes-dashboard-key-holder
    81.   namespace: kubernetes-dashboard
    82. type: Opaque
    83.  
    84. ---
    85.  
    86. kind: ConfigMap
    87. apiVersion: v1
    88. metadata:
    89.   labels:
    90.     k8s-app: kubernetes-dashboard
    91.   name: kubernetes-dashboard-settings
    92.   namespace: kubernetes-dashboard
    93.  
    94. ---
    95.  
    96. kind: Role
    97. apiVersion: rbac.authorization.k8s.io/v1
    98. metadata:
    99.   labels:
    100.     k8s-app: kubernetes-dashboard
    101.   name: kubernetes-dashboard
    102.   namespace: kubernetes-dashboard
    103. rules:
    104.   # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
    105.   - apiGroups: [""]
    106.     resources: ["secrets"]
    107.     resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    108.     verbs: ["get", "update", "delete"]
    109.     # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
    110.   - apiGroups: [""]
    111.     resources: ["configmaps"]
    112.     resourceNames: ["kubernetes-dashboard-settings"]
    113.     verbs: ["get", "update"]
    114.     # Allow Dashboard to get metrics.
    115.   - apiGroups: [""]
    116.     resources: ["services"]
    117.     resourceNames: ["heapster", "dashboard-metrics-scraper"]
    118.     verbs: ["proxy"]
    119.   - apiGroups: [""]
    120.     resources: ["services/proxy"]
    121.     resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    122.     verbs: ["get"]
    123.  
    124. ---
    125.  
    126. kind: ClusterRole
    127. apiVersion: rbac.authorization.k8s.io/v1
    128. metadata:
    129.   labels:
    130.     k8s-app: kubernetes-dashboard
    131.   name: kubernetes-dashboard
    132. rules:
    133.   # Allow Metrics Scraper to get metrics from the Metrics server
    134.   - apiGroups: ["metrics.k8s.io"]
    135.     resources: ["pods", "nodes"]
    136.     verbs: ["get", "list", "watch"]
    137.  
    138. ---
    139.  
    140. apiVersion: rbac.authorization.k8s.io/v1
    141. kind: RoleBinding
    142. metadata:
    143.   labels:
    144.     k8s-app: kubernetes-dashboard
    145.   name: kubernetes-dashboard
    146.   namespace: kubernetes-dashboard
    147. roleRef:
    148.   apiGroup: rbac.authorization.k8s.io
    149.   kind: Role
    150.   name: kubernetes-dashboard
    151. subjects:
    152.   - kind: ServiceAccount
    153.     name: kubernetes-dashboard
    154.     namespace: kubernetes-dashboard
    155.  
    156. ---
    157.  
    158. apiVersion: rbac.authorization.k8s.io/v1
    159. kind: ClusterRoleBinding
    160. metadata:
    161.   name: kubernetes-dashboard
    162. roleRef:
    163.   apiGroup: rbac.authorization.k8s.io
    164.   kind: ClusterRole
    165.   name: kubernetes-dashboard
    166. subjects:
    167.   - kind: ServiceAccount
    168.     name: kubernetes-dashboard
    169.     namespace: kubernetes-dashboard
    170.  
    171. ---
    172.  
    173. kind: Deployment
    174. apiVersion: apps/v1
    175. metadata:
    176.   labels:
    177.     k8s-app: kubernetes-dashboard
    178.   name: kubernetes-dashboard
    179.   namespace: kubernetes-dashboard
    180. spec:
    181.   replicas: 1
    182.   revisionHistoryLimit: 10
    183.   selector:
    184.     matchLabels:
    185.       k8s-app: kubernetes-dashboard
    186.   template:
    187.     metadata:
    188.       labels:
    189.         k8s-app: kubernetes-dashboard
    190.     spec:
    191.       containers:
    192.         - name: kubernetes-dashboard
    193.           image: kubernetesui/dashboard:v2.0.4
    194.           imagePullPolicy: IfNotPresent
    195.           ports:
    196.             - containerPort: 8443
    197.               protocol: TCP
    198.           args:
    199.             - --auto-generate-certificates
    200.             - --namespace=kubernetes-dashboard
    201.             # Uncomment the following line to manually specify Kubernetes API server Host
    202.             # If not specified, Dashboard will attempt to auto discover the API server and connect
    203.             # to it. Uncomment only if the default does not work.
    204.             # - --apiserver-host=http://my-address:port
    205.           volumeMounts:
    206.             - name: kubernetes-dashboard-certs
    207.               mountPath: /certs
    208.               # Create on-disk volume to store exec logs
    209.             - mountPath: /tmp
    210.               name: tmp-volume
    211.           livenessProbe:
    212.             httpGet:
    213.               scheme: HTTPS
    214.               path: /
    215.               port: 8443
    216.             initialDelaySeconds: 30
    217.             timeoutSeconds: 30
    218.           securityContext:
    219.             allowPrivilegeEscalation: false
    220.             readOnlyRootFilesystem: true
    221.             runAsUser: 1001
    222.             runAsGroup: 2001
    223.       volumes:
    224.         - name: kubernetes-dashboard-certs
    225.           secret:
    226.             secretName: kubernetes-dashboard-certs
    227.         - name: tmp-volume
    228.           emptyDir: {}
    229.       serviceAccountName: kubernetes-dashboard
    230.       nodeSelector:
    231.         "kubernetes.io/os": linux
    232.       # Comment the following tolerations if Dashboard must not be deployed on master
    233.       tolerations:
    234.         - key: node-role.kubernetes.io/master
    235.           effect: NoSchedule
    236.  
    237. ---
    238.  
    239. kind: Service
    240. apiVersion: v1
    241. metadata:
    242.   labels:
    243.     k8s-app: dashboard-metrics-scraper
    244.   name: dashboard-metrics-scraper
    245.   namespace: kubernetes-dashboard
    246. spec:
    247.   ports:
    248.     - port: 8000
    249.       targetPort: 8000
    250.   selector:
    251.     k8s-app: dashboard-metrics-scraper
    252.  
    253. ---
    254.  
    255. kind: Deployment
    256. apiVersion: apps/v1
    257. metadata:
    258.   labels:
    259.     k8s-app: dashboard-metrics-scraper
    260.   name: dashboard-metrics-scraper
    261.   namespace: kubernetes-dashboard
    262. spec:
    263.   replicas: 1
    264.   revisionHistoryLimit: 10
    265.   selector:
    266.     matchLabels:
    267.       k8s-app: dashboard-metrics-scraper
    268.   template:
    269.     metadata:
    270.       labels:
    271.         k8s-app: dashboard-metrics-scraper
    272.       annotations:
    273.         seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
    274.     spec:
    275.       containers:
    276.         - name: dashboard-metrics-scraper
    277.           image: kubernetesui/metrics-scraper:v1.0.6
    278.           imagePullPolicy: IfNotPresent
    279.           ports:
    280.             - containerPort: 8000
    281.               protocol: TCP
    282.           livenessProbe:
    283.             httpGet:
    284.               scheme: HTTP
    285.               path: /
    286.               port: 8000
    287.             initialDelaySeconds: 30
    288.             timeoutSeconds: 30
    289.           volumeMounts:
    290.           - mountPath: /tmp
    291.             name: tmp-volume
    292.           securityContext:
    293.             allowPrivilegeEscalation: false
    294.             readOnlyRootFilesystem: true
    295.             runAsUser: 1001
    296.             runAsGroup: 2001
    297.       serviceAccountName: kubernetes-dashboard
    298.       nodeSelector:
    299.         "kubernetes.io/os": linux
    300.       # Comment the following tolerations if Dashboard must not be deployed on master
    301.       tolerations:
    302.         - key: node-role.kubernetes.io/master
    303.           effect: NoSchedule
    304.       volumes:
    305.         - name: tmp-volume
    306.           emptyDir: {}

    获取登录token：

    kubectl describe secrets $(kubectl describe sa default -n kube-system | grep Mountable | awk 'NR == 2 {next} {print $3}') -n kube-system

    登录地址：https://nodeip:30008

    那么，使用配置文件登录呢？那个文件就是bootstrap.kubeconfig，将这个文件拷贝到桌面，然后登录 的时候选择文件即可了，文件内容应该如下：

    1. [root@master ~]# cat /opt/kubernetes/cfg/bootstrap.kubeconfig 
    2. apiVersion: v1
    3. clusters:
    4. - cluster:
    5.     certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVRDYzSGpYeFRiM3EzdGZUeEM4QjZwalUzYUVRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEl5TURneU56QXhNVFV3TUZvWERUSTNNRGd5TmpBeE1UVXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbGFXcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeitLb3pMQlVEYXNQTmxKc2lMSXoKZHRUR0M4a2tvNnVJZjVUSUNkY3pPemJyaks1TVJ4UzYrd2ZwVzNHOGFtN2J1QlcvRE1hcW15dGNlbEwxd0VpMwoxUGZTNE9oWXRlczUwWU4wMkZrZ2JCYmVBSVN3NnJ5cnRadGFxdWhZeHUwQjlOLzVuZGhETUx2ZFhFV1NYYWZrCmtWQXNnTFZ0dmNPMCtKVUt3OGE5eFJSRTkyWThZYXZ0azN4M3VBU2hTejUrS3FQZ1V6Q2x2a2N4UUpXVFBiTkUKOEpERXlaY0I0ay8za0NuOGtsREc3Um9Wck1hcHJ6Z3lNQkNVOEgzS1hsM0FJdkFYNGVQQTFOVGJzbUhWaDdCcgpmeWdLT0x5RHA3OUswbkp1c0NtY1JmRGJ4TWJMVCtNeU01Y0NFcm1LMkVnSTRuYXIyMndQQU5kemRTb1dIbDljCkp3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVZG1mUXNkMy85czVoKzl0V1dDMHhBL1htSENZd0h3WURWUjBqQkJnd0ZvQVVkbWZRc2QzLwo5czVoKzl0V1dDMHhBL1htSENZd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFJR0lPa0xDMlpxL3Y4dStNelVyCkpVRUY4SFV2QzdldkdXV3ZMMmlqd3dxTnJMWE1XYWd6UWJxcDM5UTVlRnRxdngzSWEveFZFZ0ZTTnBSRjNMNGYKN0VYUlYxRXpJVlUxeThaSnZzVXl1aFQyVENCQ3UwMkdvSWc1VGlJNzMwak5xQllMRGh6SVJKLzBadEtTQlcwaApIUEo3eGRybnZSdnY3WG9uT1dCbldBTUhJY0N0TzNLYlovbXY1VHBoTnkzWHJMSTdRaFVvWVlQSXN5N1BvUjhVCm9WVm80RkRRUDFPYXhGSzljSE1DNWNuLzFNSnRZUGpVRzg5RldEc01HbWVVZVZ1cnhsVStkVlFUMUZzOWJxanoKaDJWaHNtanFCK3RCbjVGdENOaEY5STVxYlJuMWJmTGRpQzl2QzJ3U00xSDZQVWRxeHB6ZlRaVHhSbEptdmtjZAo1ZTQ9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    6.     server: https://192.168.217.16:6443
    7.   name: kubernetes
    8. contexts:
    9. - context:
    10.     cluster: kubernetes
    11.     user: kubelet-bootstrap
    12.   name: default
    13. current-context: default
    14. kind: Config
    15. preferences: {}
    16. users:
    17. - name: kubelet-bootstrap
    18.   user:
    19.     token: c47ffb939f5ca36231d9e3121a252940

    两个地方需要特别注意，一个是token，这个是token.csv文件里定义的，一个是kubelet-bootstrap用户，可能使用config文件会登录失败，提示权限不足，

    那么，就先删除用户的角色绑定，重新绑定为admin，代码如下：

    1. kubectl delete clusterrolebindings kubelet-bootstrap
    2. kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=cluster-admin  --user=kubelet-bootstrap

  • 相关阅读:
    我参加第七届NVIDIA Sky Hackathon——训练CV模型
    Java NIO Selector 的使用
    高级深入--day23
    AI算法优缺点
    题目：2665.计数器 II
    PyQt5的笔记（中-4）
    【Python基础】 Python设计模式之单例模式介绍
    C语言——数组
    web前端网页设计期末课程大作业：旅游网页主题网站设计——三亚旅游网页设计(6个页面) HTML+CSS+JavaScript
    搞定面试官 - 可以介绍一下 MySQL InnoDB 引擎的索引模型嘛？
  • 原文地址：https://blog.csdn.net/alwaysbefine/article/details/126540138