使用 etcdadm 快速、弹性部署 etcd 集群

Etcd 是一个可靠的分布式键值存储, 常用于分布式系统关键数据的存储；而 etcdadm 是一个用于操作 etcd 集群的命令行工具，它可以轻松创建集群、向现有集群添加成员、从现有集群中删除成员等操作；其使用方式类似 kubeadm, 即主要操作流程为: 先启动第一个集群节点，后续节点直接 join 即可

建议通过 PC 端访问本文章，以获取更好阅读体验，由于精力有限，该文章的后续更新、完善仅限于站点 运维技术帮 (https://ywjsbang.com) 望理解 !!

测试环境#

节点主机名	节点 IP 地址	系统版本	etcd 版本	etcdadm 版本
c7	192.168.31.37	CentOS 7.9.2009 ( 5.4.180-1.el7 )	V3.5.5	V0.1.5
c8	192.168.31.38	同上	同上	同上
c9	192.168.31.39	同上	同上	同上

安装 etcdadm#

1、预编译二进制安装

wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.5/etcdadm-linux-amd64
mv etcdadm-linux-amd64 /usr/local/bin/etcdadm
chmod +x /usr/local/bin/etcdadm
scp /usr/local/bin/etcdadm 192.168.31.{38,39}:/usr/local/bin/

2、各节点系统防火墙放行端口 2379，2380

firewall-cmd --add-port=2379/tcp
firewall-cmd --add-port=2380/tcp

初始化 etcd 节点#

1、初始化第一个 etcd 集群节点

etcdadm init \
--version "3.5.5" \
--init-system "systemd" \
--install-dir "/opt/bin/" \
--certs-dir "/etc/etcd/pki" \
--data-dir "/var/lib/etcd" \
--release-url "https://github.com/etcd-io/etcd/releases/download"

# 主要选项解析
--version      # 指定部署的 etcd 版本
--init-system  # 设置 etcd 进程管理方式，默认 systemd，取值 kubelet 时，则以容器方法运行 etcd 进程
--install-dir  # etcd 二进制程序安装目录

2、etcdadm init 初始化过程解析

# 下载解压、安装二进制文件 etcd、etcdctl
2022-10-20 14:26:12.781166 I | [install] Artifact not found in cache. Trying to fetch from upstream: https://github.com/etcd-io/etcd/releases/download
INFO[0000] [install] Downloading & installing etcd https://github.com/etcd-io/etcd/releases/download from 3.5.5 to /var/cache/etcdadm/etcd/v3.5.5
INFO[0000] [install] downloading etcd from https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz
INFO[0009] [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz to /tmp/etcd641204404
INFO[0009] [install] verifying etcd 3.5.5 is installed in /opt/bin/

# 生成一个自签名的 CA 证书及私钥
INFO[0001] [certificates] creating PKI assets
INFO[0001] creating a self signed etcd CA certificate and key files
[certificates] Generated ca certificate and key.
> /etc/etcd/pki/ca.crt
> /etc/etcd/pki/ca.key

# 生成一个 server 证书及私钥
INFO[0001] creating a new server certificate and key files for etcd
[certificates] Generated server certificate and key.
[certificates] server serving cert is signed for DNS names [c7] and IPs [192.168.31.37 127.0.0.1]
# > /etc/etcd/pki/server.crt
# > /etc/etcd/pki/server.key

# 生成一个 peer 证书及私钥
INFO[0001] creating a new certificate and key files for etcd peering
[certificates] Generated peer certificate and key.
[certificates] peer serving cert is signed for DNS names [c7] and IPs [192.168.31.37]
# > /etc/etcd/pki/peer.crt
# > /etc/etcd/pki/peer.key

# 生成一个用于 etcdctl 的 client 证书及私钥
INFO[0001] creating a new client certificate for the etcdctl
[certificates] Generated etcdctl-etcd-client certificate and key.
# > /etc/etcd/pki/etcdctl-etcd-client.crt
# > /etc/etcd/pki/etcdctl-etcd-client.key

# 生成一个用于 k8s apiserver 调用 etcd 时的 client 证书及私钥
INFO[0002] creating a new client certificate for the apiserver calling etcd
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"
# > /etc/etcd/pki/apiserver-etcd-client.crt
# > /etc/etcd/pki/apiserver-etcd-client.key

# 检查本地 etcd 端点是否健康
INFO[0003] [health] Checking local etcd endpoint health
INFO[0003] [health] Local etcd endpoint is healthy

# 复制 CA cert/key 到其它 etcd 节点，并在其它 etcd 节点运行 etcdadm join 命令， 将其它 etcd 节点加入集群
INFO[0003] To add another member to the cluster, copy the CA cert/key to its certificate dir and run:
INFO[0003]  etcdadm join https://192.168.31.37:2379

3、向其它节点分发 CA 根证书及私钥

ssh root@192.168.31.38 "mkdir /etc/etcd/pki/"
scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.38:/etc/etcd/pki/

ssh root@192.168.31.39 "mkdir /etc/etcd/pki/"
scp -r /etc/etcd/pki/{ca.crt,ca.key} 192.168.31.39:/etc/etcd/pki/

添加 etcd 节点#

若当前主机无法下载，可提前将 etcd 二进制程序包存放在如下路径: /var/cache/etcdadm/etcd/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz

1、添加节点 192.168.31.38

etcdadm join https://192.168.31.38:2379 \
--version "3.5.5" \
--init-system "systemd" \
--install-dir "/opt/bin/"   \
--certs-dir  "/etc/etcd/pki" \
--data-dir "/var/lib/etcd"  \
--release-url "https://github.com/etcd-io/etcd/releases/download"

2、添加节点 192.168.31.39

etcdadm join https://192.168.31.38:2379 \
--version "3.5.5" \
--init-system "systemd" \
--install-dir "/opt/bin/"   \
--certs-dir  "/etc/etcd/pki" \
--data-dir "/var/lib/etcd"  \
--release-url "https://github.com/etcd-io/etcd/releases/download"

Etcd Server#

1、用于 Etcd Server 的环境变量配置 /etc/etcd/etcd.env

ETCD_NAME=c7

# Initial cluster configuration
ETCD_INITIAL_CLUSTER=c7=https://192.168.31.37:2380
ETCD_INITIAL_CLUSTER_TOKEN=dee8095f
ETCD_INITIAL_CLUSTER_STATE=new

# Peer configuration
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.31.37:2380
ETCD_LISTEN_PEER_URLS=https://192.168.31.37:2380

ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Client/server configuration
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.31.37:2379
ETCD_LISTEN_CLIENT_URLS=https://192.168.31.37:2379,https://127.0.0.1:2379

ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_CERT_FILE=/etc/etcd/pki/server.crt
ETCD_KEY_FILE=/etc/etcd/pki/server.key
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Other
ETCD_DATA_DIR=/var/lib/etcd
ETCD_STRICT_RECONFIG_CHECK=true
GOMAXPROCS=8

# Logging configuration
# Profiling/metrics

2、Etcd Server 启动脚本

# cat /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd-member.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd/etcd.env
ExecStart=/opt/bin/etcd

Type=notify
TimeoutStartSec=0
Restart=on-failure
RestartSec=5s

LimitNOFILE=65536
Nice=-10
IOSchedulingClass=best-effort
IOSchedulingPriority=2
MemoryLow=200M

[Install]
WantedBy=multi-user.target

etcdctl.sh#

1、用于 etcdctl 的环境变量配置 /etc/etcd/etcdctl.env

export ETCDCTL_API=3

export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/pki/etcdctl-etcd-client.crt
export ETCDCTL_KEY=/etc/etcd/pki/etcdctl-etcd-client.key

export ETCDCTL_DIAL_TIMEOUT=3s

2、脚本 etcdctl.sh 是对 etcdctl 命令的简单包装，其用法与 etcdctl 一致

cat /opt/bin/etcdctl.sh
#!/usr/bin/env sh
if ! [ -r "/etc/etcd/etcdctl.env" ]; then
        echo "Unable to read the etcdctl environment file '/etc/etcd/etcdctl.env'. The file must exist, and this wrapper must be run as root."
        exit 1
fi
. "/etc/etcd/etcdctl.env"   # 相当于 source 该环境变量配置文件
"/opt/bin/etcdctl" "$@"     # $@ 表示脚本 etcdctl.sh 的命令行参数

管理命令#

# 查看命令行 init 或 join 的帮助信息
etcdadm init|join --help

# 从 etcd 集群移除当前节点
etcdadm reset

# 查看集群节点成员
/opt/bin/etcdctl.sh member list
# > 19fc11a542653f62, started, c9, https://192.168.31.39:2380, https://192.168.31.39:2379, false
# > 9a246c6786d36273, started, c7, https://192.168.31.37:2380, https://192.168.31.37:2379, false
# > a509d3d8e8aa4911, started, c8, https://192.168.31.38:2380, https://192.168.31.38:2379, false

# 查看当前节点是否正常
/opt/bin/etcdctl.sh endpoint health
# 127.0.0.1:2379 is healthy: successfully committed proposal: took = 17.112587ms

# 查看当前节点状态
/opt/bin/etcdctl.sh endpoint status
# > 127.0.0.1:2379, 9a246c6786d36273, 3.5.5, 20 kB, true, false, 3, 10, 10,

由于笔者时间、视野、认知有限，本文难免出现错误、疏漏等问题，期待各位读者朋友、业界大佬指正交流, 共同进步 !!