ETCD集群搭建（实践可用）

概述

    etcd 是兼具一致性和高可用性的键值数据库，可以作为保存 Kubernetes 所有集群数据的后台数据库。

- 官方网址：

    Documentation versions | etcd

准备cfssl证书生成工具

    cfssl是一个开源的证书管理工具，使用json文件生成证书.

在任意一台服务器上操作，这里选择k8s01

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

创建etcd相关目录

$ mkdir -pv /opt/kubernetes/etcd/{bin,cfg,ssl,data}

创建证书

4.1 创建ca证书json文件

$ cd /opt/kubernetes/etcd/ssl

# 创建ca-config

$ vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

# 创建ca-csr

$ vim ca-csr.json 
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}

4.2 生成ca证书

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

4.3 使用自签ca签发etcd证书

创建证书申请文件：

$ cd /opt/kubernetes/etcd/ssl

# 注意hosts内容，etcd集群内的ip都要写上，可以预留几个，为以后扩容使用

{
    "CN": "etcd",
    "hosts": [
    "10.10.21.73",
    "10.10.21.74",
    "10.10.21.75"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

生成证书

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# 会生成server.pem和server-key.pem文件 

$ ll |grep server
-rw-r--r-- 1 root root 1013 Sep 14 15:06 server.csr
-rw-r--r-- 1 root root  290 Sep 14 15:05 server-csr.json
-rw------- 1 root root 1679 Sep 14 15:06 server-key.pem
-rw-r--r-- 1 root root 1338 Sep 14 15:06 server.pem

下载etcd二进制文件

下载地址

https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz

部署ETCD集群

6.1 复制二进制文件到指定文件

tar xf etcd-v3.5.0-linux-amd64.tar.gz
cp etcd-v3.5.0-linux-amd64/{etcd,etcdctl,etcdutl} /opt/kubernetes/etcd/bin

6.2 创建etcd配置文件

$vim /opt/kubernetes/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1"    # k8s01为etcd-1，k8s02为etcd-2。。。每个节点唯一标识符
ETCD_DATA_DIR="/opt/kubernetes/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.241:2380"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.241:2379"    # 修改对应ip，k8s01为241，k8s02为242...
 
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.241:2380"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.241:2379"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.241:2380,etcd-2=https://192.168.1.242:2380,etcd-3=https://192.168.1.243:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

注释：

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIALCLUSTER_TOKEN：集群Token

ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

6.3 创建systemd文件

$ vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.targe
 
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/etcd/cfg/etcd.conf
ExecStart=/opt/kubernetes/etcd/bin/etcd \
--cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--peer-key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

分发etcd文件

node02

$ scp -r /opt/kubernetes k8s02:/opt/
$ scp /etc/systemd/system/etcd.service k8s02:/etc/systemd/system/
# 记得修改etcd配置文件

node03

$ scp -r /opt/kubernetes k8s03:/opt/
$ scp /etc/systemd/system/etcd.service k8s03:/etc/systemd/system/
# 记得修改etcd配置文件

8. 分别启动etcd服务

$ systemctl start etcd.service

查看集群状态

[root@kubenode01 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" endpoint health --write-out=table
 
+--------------------------+--------+-------------+-------+
|         ENDPOINT         | HEALTH |    TOOK     | ERROR |
+--------------------------+--------+-------------+-------+
| https://10.10.21.75:2379 |   true | 13.407895ms |       |
| https://10.10.21.74:2379 |   true |  13.61133ms |       |
| https://10.10.21.73:2379 |   true | 14.868649ms |       |
+--------------------------+--------+-------------+-------+

9. 测试

写入数据

[root@kubenode01 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" put foo bar
OK

读取数据

[root@kubenode02 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" get foo
foo
bar