• 离线安装 K3S


    一、前言

    简要记录一下离线环境下 K3S 的搭建,版本为 v1.23.17+k3s1,使用外部数据库 MySQL
    元数据存储,禁用默认组件(corednsservicelbtraefiklocal-storagemetrics-server)并使用 Helm
    单独安装(corednsmetrics-servertraefiklonghorn)。

    需要一台联网主机(虚拟机),和多台未联网主机(服务器)。

    二、联网虚拟机

    2.1 快速引导一个单节点集群

    curl -fsSL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.23.17+k3s1 bash -s - server \
        --data-dir /data/k3s/var/lib/rancher/k3s \
        --cluster-cidr 10.8.0.0/16 \
        --service-cidr 10.16.0.0/16 \
        --cluster-dns 10.16.0.10 \
        --service-node-port-range 1-65535 \
        --kube-proxy-arg proxy-mode=ipvs \
        --disable coredns \
        --disable servicelb \
        --disable traefik \
        --disable local-storage \
        --disable metrics-server
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12

    2.2 快速安装 Longhorn 的依赖

    Installation Requirements

    # yum
    yum install iscsi-initiator-utils nfs-utils
    
    # ubuntu
    apt install open-iscsi nfs-common
    
    # 启动
    systemctl enable iscsid --now
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8

    2.3 快速安装应用(通过 Helm Controller)

    ### coredns
    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: coredns
      namespace: kube-system
      labels:
        app: coredns
    spec:
      repo: https://coredns.github.io/helm
      chart: coredns
      targetNamespace: kube-system
      bootstrap: true
      valuesContent: |-
        fullnameOverride: coredns
        serviceType: ClusterIP
        service:
          clusterIP: 10.16.0.10
          name: coredns
        servers:
          - zones:
              - zone: .
            port: 53
            plugins:
              - name: errors
              - name: health
                configBlock: |-
                  lameduck 5s
              - name: ready
              - name: kubernetes
                parameters: cluster.local in-addr.arpa ip6.arpa
                configBlock: |-
                  pods insecure
                  fallthrough in-addr.arpa ip6.arpa
                  ttl 30
              - name: prometheus
                parameters: 0.0.0.0:9153
              - name: forward
                parameters: . /etc/resolv.conf
              - name: cache
                parameters: 30
              - name: loop
              - name: reload
              - name: loadbalance
    
    
    ###  metrics-server
    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: metrics-server
      namespace: kube-system
      labels:
        app: metrics-server
    spec:
      repo: https://charts.bitnami.com/bitnami
      chart: metrics-server
      targetNamespace: kube-system
      bootstrap: true
      valuesContent: |
        apiService:
          create: true
        extraArgs:
          - --kubelet-insecure-tls
          - --kubelet-use-node-status-port
          - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
          - --metric-resolution=15s
    
    
    ### traefik
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: traefik-system
    
    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: traefik
      namespace: traefik-system
      labels:
        app: traefik
    spec:
      repo: https://traefik.github.io/charts
      chart: traefik
      targetNamespace: traefik-system
      bootstrap: true
      valuesContent: |-
        deployment:
          kind: Deployment
        ingressClass:
          enabled: true
          isDefaultClass: true
        providers:
          kubernetesCRD:
            enabled: true
            allowCrossNamespace: true
            allowExternalNameServices: true
            allowEmptyServices: true
          kubernetesIngress:
            enabled: true
            allowExternalNameServices: true
            allowEmptyServices: true
            publishedService:
              enabled: true
        ports:
          traefik:
            port: 9000
            protocol: TCP
            expose: false
            exposedPort: 9000
          metrics:
            port: 9100
            protocol: TCP
            expose: false
            exposedPort: 9100
          web:
            port: 80
            protocol: TCP
            expose: true
            exposedPort: 80
            nodePort: 30080
          websecure:
            port: 443
            protocol: TCP
            expose: true
            exposedPort: 443
            ndoePort: 30443
            tls:
              enabled: true
        service:
          type: NodePort
        securityContext:
          capabilities:
            drop: []
            add: [ALL]
          readOnlyRootFilesystem: false
        podSecurityContext:
          runAsGroup: 0
          runAsNonRoot: false
          runAsUser: 0
    
    
    ### longhorn
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: longhorn-system
    
    ---
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: longhorn
      namespace: longhorn-system
      labels:
        app: longhorn
    spec:
      repo: https://charts.longhorn.io
      chart: longhorn
      targetNamespace: longhorn-system
      bootstrap: true
      valuesContent: |-
        persistence:
          defaultClassReplicaCount: 1
        csi:
          attacherReplicaCount: 1
          provisionerReplicaCount: 1
          resizerReplicaCount: 1
          snapshotterReplicaCount: 1
        defaultSettings:
          defaultDataPath: /data/longhorn
          defaultReplicaCount: 1
          deletingConfirmationFlag: true
        longhornUI:
          replicas: 1
        longhornConversionWebhook:
          replicas: 1
        longhornAdmissionWebhook:
          replicas: 1
        longhornRecoveryBackend:
          replicas: 1
        ingress:
          enabled: true
          host: longhorn.example.org
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
    • 63
    • 64
    • 65
    • 66
    • 67
    • 68
    • 69
    • 70
    • 71
    • 72
    • 73
    • 74
    • 75
    • 76
    • 77
    • 78
    • 79
    • 80
    • 81
    • 82
    • 83
    • 84
    • 85
    • 86
    • 87
    • 88
    • 89
    • 90
    • 91
    • 92
    • 93
    • 94
    • 95
    • 96
    • 97
    • 98
    • 99
    • 100
    • 101
    • 102
    • 103
    • 104
    • 105
    • 106
    • 107
    • 108
    • 109
    • 110
    • 111
    • 112
    • 113
    • 114
    • 115
    • 116
    • 117
    • 118
    • 119
    • 120
    • 121
    • 122
    • 123
    • 124
    • 125
    • 126
    • 127
    • 128
    • 129
    • 130
    • 131
    • 132
    • 133
    • 134
    • 135
    • 136
    • 137
    • 138
    • 139
    • 140
    • 141
    • 142
    • 143
    • 144
    • 145
    • 146
    • 147
    • 148
    • 149
    • 150
    • 151
    • 152
    • 153
    • 154
    • 155
    • 156
    • 157
    • 158
    • 159
    • 160
    • 161
    • 162
    • 163
    • 164
    • 165
    • 166
    • 167
    • 168
    • 169
    • 170
    • 171
    • 172
    • 173
    • 174
    • 175
    • 176
    • 177
    • 178
    • 179
    • 180
    • 181
    • 182
    • 183
    • 184
    • 185
    • 186
    • 187
    • 188
    • 189
    • 190
    kubectl apply -f charts.yaml
    
    • 1

    三、资源准备

    3.1 下载 Longhorn 依赖

    1. 查看服务器 glibc 版本

      ldd --version
      
      • 1
      osos versionglibc version
      centos7.92.17
      centos8.42.28
      ubuntu18.042.27
      ubuntu20.042.31
      ubuntu22.042.35
    2. 创建对应 glibc 版本的容器

      # centos 7
      kubectl run centos --image=centos:7.9.2009 --command -- /bin/sleep infinity
      kubectl exec -it pod/centos -- /bin/bash
      
      # ubuntu 22
      kubectl run ubuntu --image=ubuntu:22.04 --command -- /bin/sleep infinity
      kubectl exec -it pod/ubuntu -- /bin/bash
      
      • 1
      • 2
      • 3
      • 4
      • 5
      • 6
      • 7
    3. 下载依赖

      # yum
      yum install iscsi-initiator-utils nfs-utils --downloadonly --downloaddir=rpm -y
      tar -czvf ./rpm.tar.gz ./rpm
      
      # apt
      apt update && apt install open-iscsi nfs-common --download-only -y && mkdir -p deb && cp /var/cache/apt/archives/*.deb deb
      tar -czvf ./deb.tar.gz ./deb
      
      • 1
      • 2
      • 3
      • 4
      • 5
      • 6
      • 7
    4. 复制出依赖

      # yum
      kubectl cp centos:/rpm.tar.gz ./rpm.tar.gz
      
      # apt
      kubectl cp ubuntu:/deb.tar.gz ./deb.tar.gz
      
      • 1
      • 2
      • 3
      • 4
      • 5

    3.2 下载 K3S 资源

    参考文档:离线安装

    wget https://github.com/k3s-io/k3s/releases/download/v1.23.17+k3s1/k3s-airgap-images-amd64.tar.gz
    wget https://github.com/k3s-io/k3s/releases/download/v1.23.17+k3s1/k3s
    wget https://get.helm.sh/helm-v3.12.2-linux-amd64.tar.gz
    wget https://get.k3s.io -O install.sh
    
    • 1
    • 2
    • 3
    • 4

    3.3 下载 HelmChart 和导出镜像

    # 下载 helm chart 包
    helm repo add coredns  https://coredns.github.io/helm     && helm pull coredns/coredns        --version 1.26.0
    helm repo add bitnami  https://charts.bitnami.com/bitnami && helm pull bitnami/metrics-server --version 6.5.2
    helm repo add traefik  https://traefik.github.io/charts   && helm pull traefik/traefik        --version 24.0.0
    helm repo add longhorn https://charts.longhorn.io         && helm pull longhorn/longhorn      --version 1.5.1
    
    # 导出镜像
    k3s ctr image ls -q | grep -v 'sha256' | sort -u | xargs k3s ctr image export image.tar
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8

    四、未联网服务器

    4.1 准备 K3S 资源

    # 准备 k3s 镜像
    mkdir -p /data/k3s/var/lib/rancher/k3s/agent/images
    cp ./k3s-airgap-images-amd64.tar.gz /data/k3s/var/lib/rancher/k3s/agent/images
    
    # 准备 k3s 二进制可执行文件
    install ./k3s /usr/local/bin
    
    # 准备 helm 二进制可执行文件
    tar -zxvf ./helm-v3.12.2-linux-amd64.tar.gz
    install ./linux-amd64/helm /usr/local/bin
    
    # 准备 k3s 安装脚本
    chmod +x ./install.sh
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13

    4.2 引导第一个 Server 节点启动

    # 引导 Server
    INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.23.17+k3s1 ./install.sh server \
        --data-dir /data/k3s/var/lib/rancher/k3s \
        --cluster-cidr 10.8.0.0/16 \
        --service-cidr 10.16.0.0/16 \
        --cluster-dns 10.16.0.10 \
        --service-node-port-range 1-65535 \
        --kube-proxy-arg proxy-mode=ipvs \
        --disable coredns \
        --disable servicelb \
        --disable traefik \
        --disable local-storage \
        --disable metrics-server \
        --datastore-endpoint="mysql://:@tcp(:3306)/"
    
    # 查看 Token
    cat /data/k3s/var/lib/rancher/k3s/server/token
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17

    4.3 引导其它 Server 节点加入

    配置标识在所有 Server 节点必须是相同的。

    INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.23.17+k3s1 ./install.sh server \
        --data-dir /data/k3s/var/lib/rancher/k3s \
        --cluster-cidr 10.8.0.0/16 \
        --service-cidr 10.16.0.0/16 \
        --cluster-dns 10.16.0.10 \
        --service-node-port-range 1-65535 \
        --kube-proxy-arg proxy-mode=ipvs \
        --disable coredns \
        --disable servicelb \
        --disable traefik \
        --disable local-storage \
        --disable metrics-server \
        --datastore-endpoint="mysql://:@tcp(:3306)/" \
        --token <TOKEN>
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14

    4.4 引导 Agent 节点加入

    INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.23.17+k3s1 ./install.sh server \
        --data-dir /data/k3s/var/lib/rancher/k3s \
        --datastore-endpoint="mysql://:@tcp(:3306)/" \
        --token <TOKEN>
    
    • 1
    • 2
    • 3
    • 4

    4.5 安装 Longhorn 依赖

    # yum
    tar -zxvf rpm.tar.gz
    rpm -ivh ./rpm/*.rpm
    
    # apt
    tar -zxvf deb.tar.gz
    apt install ./deb/*.deb
    
    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7

    4.4 导入镜像和安装应用

    # 导出镜像
    k3s ctr image import ./image.tar
    
    # coredns
    helm install coredns coredns-1.26.0.tgz --namespace kube-system --values <VALUES_YAML_FILE>
    
    • 1
    • 2
    • 3
    • 4
    • 5
  • 相关阅读:
    Avoiding Row-by-Row Processing 避免逐行处理
    阿里巴巴Java方向面试题汇总(含答案)
    根据你的工作经历,说说软件测试中质量体系建设
    Spring事务的概念(四大特性)
    【Gateway】基于ruoyi-cloud-plus项目,gateway局部过滤器和过滤返回以及集成nacos
    【ONLYOFFICE震撼8.1】ONLYOFFICE8.1版本桌面编辑器测评
    计算机视觉 Project 1:Image Filtering and Hybrid Images
    NX二次开发:保存时导出PDF并打开
    【GoWeb项目-个人Blog】初始化数据库和日志
    【从0到1设计一个网关】什么是网关?以及为什么需要自研网关?
  • 原文地址:https://blog.csdn.net/XY1790026787/article/details/133226047