Centos7 部署 Containerd

        最近工作中需要部署Containerd给kubernetes集群作为运行时容器，特此记录便于日后查阅

Containerd Github地址：https://containerd.io/downloads/

        Containerd安装我们使用 1.6.4 版本号

containerd-1.6.4-linux-amd64.tar.gz 只包含containerd
cri-containerd-cni-1.6.4-linux-amd64.tar.gz 包含containerd以及cri runc等相关工具包，建议下载本包

#下载tar.gz包
#containerd工具包，包含cri runc等
wget https://github.com/containerd/containerd/releases/download/v1.6.4/cri-containerd-cni-1.6.4-linux-amd64.tar.gz
 
#备用下载地址
wget https://d.frps.cn/file/kubernetes/containerd/cri-containerd-cni-1.6.4-linux-amd64.tar.gz

        也可以选择直接下载我上传的安装包，传送门：

        Kubernetes CRI Containerd 1.6.4 运行时容器安装包下载地址

        工具包文件如下

#cri-containerd-cni会将我们整个containerd相关的依赖都进行下载下来
 
[root@k8s-01 containerd]# tar zxvf cri-containerd-cni-1.6.4-linux-amd64.tar.gz -C /   #我们直接让它给我们对应的目录给替换掉
etc/
etc/systemd/
etc/systemd/system/
etc/systemd/system/containerd.service
etc/crictl.yaml
etc/cni/
etc/cni/net.d/
etc/cni/net.d/10-containerd-net.conflist
usr/
usr/local/
usr/local/sbin/
usr/local/sbin/runc
usr/local/bin/
usr/local/bin/crictl
usr/local/bin/ctd-decoder
usr/local/bin/ctr
usr/local/bin/containerd-shim
usr/local/bin/containerd
usr/local/bin/containerd-shim-runc-v1
usr/local/bin/critest
usr/local/bin/containerd-shim-runc-v2
usr/local/bin/containerd-stress
opt/
opt/containerd/
opt/containerd/cluster/
opt/containerd/cluster/version
opt/containerd/cluster/gce/
opt/containerd/cluster/gce/cni.template
opt/containerd/cluster/gce/env
opt/containerd/cluster/gce/configure.sh
opt/containerd/cluster/gce/cloud-init/
opt/containerd/cluster/gce/cloud-init/node.yaml
opt/containerd/cluster/gce/cloud-init/master.yaml
opt/cni/
opt/cni/bin/
opt/cni/bin/firewall
opt/cni/bin/portmap
opt/cni/bin/host-local
opt/cni/bin/ipvlan
opt/cni/bin/host-device
opt/cni/bin/sbr
opt/cni/bin/vrf
opt/cni/bin/static
opt/cni/bin/tuning
opt/cni/bin/bridge
opt/cni/bin/macvlan
opt/cni/bin/bandwidth
opt/cni/bin/vlan
opt/cni/bin/dhcp
opt/cni/bin/loopback
opt/cni/bin/ptp

上面的文件都是二进制文件，直接移动到对应的目录并配置好环境变量就可以进行使用了

        如果我们机器上通过yum安装docker了，可以用下面的命令进行卸载

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

        接下来我们为每台服务器配置Containerd

#创建配置文件目录
[root@k8s-01 ~]# mkdir /etc/containerd -p
 
#生成默认配置文件
[root@k8s-01 ~]# containerd config default > /etc/containerd/config.toml
 
#--config,-c可以在启动守护程序时更改此路径
#配置文件的默认路径位于/etc/containerd/config.toml

        替换默认pause镜像地址

        默认情况下k8s.gcr.io无法访问，所以使用我提供的阿里云镜像仓库地址即可

sed -i 's/k8s.gcr.io/registry.cn-beijing.aliyuncs.com\/abcdocker/' /etc/containerd/config.toml 
 
#所有节点更换默认镜像地址
#我这里使用阿里云地址

        配置systemd作为容器的cgroup driver

sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/' /etc/containerd/config.toml

        Containerd官方操作手册

        默认cri-containerd-cni包中会有containerd启动脚本，我们已经解压到对应的目录，可以直接调用启动

[root@k8s-01 ~]# systemctl enable containerd --now   
Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /etc/systemd/system/containerd.service.
 
[root@k8s-01 ~]# systemctl status containerd   #查看containerd启动状态
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-05-12 22:59:19 EDT; 3s ago
     Docs: https://containerd.io
  Process: 30048 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 30050 (containerd)
   Memory: 24.5M
   CGroup: /system.slice/containerd.service
           └─30050 /usr/local/bin/containerd
 
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.153514446-04:00" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154085898-04:00" level=info msg="Start subscribing containerd event"
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154137039-04:00" level=info msg="Start recovering state"
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154230615-04:00" level=info msg="Start event monitor"
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154276701-04:00" level=info msg="Start snapshots syncer"
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154299287-04:00" level=info msg="Start cni network conf syncer for default"
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154316094-04:00" level=info msg="Start streaming server"
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154675632-04:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.154755704-04:00" level=info msg=serving... address=/run/containerd/containerd.sock
May 12 22:59:19 web01 containerd[30050]: time="2022-05-12T22:59:19.155220379-04:00" level=info msg="containerd successfully booted in 0.027654s"

         ctr在我们解压包中已经附带了，直接可以使用

[root@k8s-01 ~]# ctr version
Client:		#ctr版本号
  Version:  v1.6.4
  Revision: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
  Go version: go1.17.9
 
Server:
  Version:  v1.6.4     #containerd版本号
  Revision: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
  UUID: b376d7b6-c97e-4b39-8144-9624ade3ba84
 
#可以使用下面命令查看containerd版本号
[root@k8s-01 ~]# containerd --version
containerd github.com/containerd/containerd v1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16

        指定 kubernetes 使用 containerd 作为运行时容器，首先查看kubelet的service环境变量内容：

$ vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

         修改 /var/lib/kubelet/kubeadm-flags.env 环境变量文件，指定Containerd，内容如下：

$ /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7"

        如果是首次启动Kubernetes集群，需要通过kubeadm拉起kubelet（不能单独启动kubelet），否则不能生成环境配置文件导致kubelet启动失败

        master拉起方法：

$ kubeadm init --config=init.default.yaml
# 重置 如果有需要
$ kubeadm reset

        slave拉起方法：

$ kubeadm join 10.0.61.12:6443 --token wxf9yd.teut5tqzrl6ddwsu     --discovery-token-ca-cert-hash sha256:d94b6a510d55f634996257f2e9ed95caae6153c44cf013ffbc261c03a4f5754a
# 重置 如果有需要
$ kubeadm reset

        指定Harbor镜像仓库，内容如下：

$ vim /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
 
[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
 
[ttrpc]
  address = ""
  uid = 0
  gid = 0
 
[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""
 
[metrics]
  address = ""
  grpc_histogram = false
 
[cgroup]
  path = ""
 
[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"
 
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.5"
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      max_conf_num = 1
      conf_template = ""
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://hub-mirror.c.163.com/", "https://registry-1.docker.io"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.0.61.99:30006"]
          endpoint = ["http://10.0.61.99:30006"]

        到此 Centos7 部署 Containerd 介绍完成。