K8S集群配置私有仓库和集群功能演示

配置私有仓库（master/node1/node2上操作）

编辑/etc/docker/daemon.json文件，添加insecure-registries参数，告诉私有仓库是安全的

vim /etc/docker/daemon.json
1

daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
  "max-size": "100m"
  },
  "insecure-registries": ["https://hub.atguigu.com"]
  
}
1
2
3
4
5
6
7
8
9

重启docker，查看状态

systemctl daemon-reload
systemctl restart docker
systemctl status docker
1
2
3

---

安装harbor（相关文件开头链接出下载，harbor服务器上操作）

升级内核

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm 
# 安装完成后检查 /boot/grub2/grub.cfg 中对应内核 menuentry 中是否包含 initrd16 配置，如果没有，再安装 一次！ 
yum --enablerepo=elrepo-kernel install -y kernel-lt 

#查看系统内部内核数
cat /boot/grub2/grub.cfg |grep menuentry

# 设置开机从新内核启动 ，具体可以从grub.cfg的文件中获取
grub2-set-default 'CentOS Linux (5.4.207-1.el7.elrepo.x86_64) 7 (Core)'

#查看默认启动项
grub2-editenv list

1
2
3
4
5
6
7
8
9
10
11
12
13

安装docker

yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum update -y && yum install -y docker-ce

## 创建 /etc/docker 目录
mkdir /etc/docker

# 配置 daemon.json，设置cgroup的管理方式为systemd
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
	"max-size": "100m"
  }
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# 重启docker服务
systemctl daemon-reload && systemctl restart docker && systemctl enable docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

准备docker-compose和harbor-offline-installer-v1.2.0.tgz文件
安装xshell指令用于传输文件

yum -y install lrzsz
1

安装docker-compose

# 文件到移动docker-compose文件到/usr/local/bin/目录下
mv docker-compose /usr/local/bin/
# 授权
chmod a+x /usr/local/bin/docker-compose 
# 查看版本信息，校验是否安装成功
docker-compose version
1
2
3
4
5
6

解压harbor文件

tar -zxvf harbor-offline-installer-v1.2.0.tgz
mv harbor /usr/local/
cd /usr/local/harbor/
1
2
3

编辑harbor.cfg文件,修改域名和https方式

vi harbor.cfg
1

harbor.cfg

## Configuration file of Harbor

#The IP address or hostname to access admin UI and registry service.
#DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname = hub.atguigu.com

#The protocol for accessing the UI and token/notification service, by default it is http.
#It can be set to https if ssl is enabled on nginx.
ui_url_protocol = https

#The password for the root user of mysql db, change this before any production use.
db_password = root123

#Maximum number of job workers in job service  
max_job_workers = 3 

#Determine whether or not to generate certificate for the registry's token.
#If the value is on, the prepare script creates new root cert and private key 
#for generating token to access the registry. If the value is off the default key/cert will be used.
#This flag also controls the creation of the notary signer's cert.
customize_crt = on

#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert = /data/cert/server.crt
ssl_cert_key = /data/cert/server.key

#The path of secretkey storage
secretkey_path = /data

#Admiral's url, comment this attribute, or set its value to NA when Harbor is standalone
admiral_url = NA

#The password of the Clair's postgres database, only effective when Harbor is deployed with Clair.
#Please update it before deployment, subsequent update will cause Clair's API server and Harbor unable to access Clair's database.
clair_db_password = password

#NOTES: The properties between BEGIN INITIAL PROPERTIES and END INITIAL PROPERTIES
#only take effect in the first boot, the subsequent changes of these properties 
#should be performed on web ui

#************************BEGIN INITIAL PROPERTIES************************

#Email account settings for sending out password resetting emails.

#Email server uses the given username and password to authenticate on TLS connections to host and act as identity.
#Identity left blank to act as username.
email_identity = 

email_server = smtp.mydomain.com
email_server_port = 25
email_username = sample_admin@mydomain.com
email_password = abc
email_from = admin <sample_admin@mydomain.com>
email_ssl = false

##The initial password of Harbor admin, only works for the first time when Harbor starts. 
#It has no effect after the first launch of Harbor.
#Change the admin password from UI after launching Harbor.
harbor_admin_password = Harbor12345

##By default the auth mode is db_auth, i.e. the credentials are stored in a local database.
#Set it to ldap_auth if you want to verify a user's credentials against an LDAP server.
auth_mode = db_auth

#The url for an ldap endpoint.
ldap_url = ldaps://ldap.mydomain.com

#A user's DN who has the permission to search the LDAP/AD server. 
#If your LDAP/AD server does not support anonymous search, you should configure this DN and ldap_search_pwd.
#ldap_searchdn = uid=searchuser,ou=people,dc=mydomain,dc=com

#the password of the ldap_searchdn
#ldap_search_pwd = password

#The base DN from which to look up a user in LDAP/AD
ldap_basedn = ou=people,dc=mydomain,dc=com

#Search filter for LDAP/AD, make sure the syntax of the filter is correct.
#ldap_filter = (objectClass=person)

# The attribute used in a search to match a user, it could be uid, cn, email, sAMAccountName or other attributes depending on your LDAP/AD  
ldap_uid = uid 

#the scope to search for users, 1-LDAP_SCOPE_BASE, 2-LDAP_SCOPE_ONELEVEL, 3-LDAP_SCOPE_SUBTREE
ldap_scope = 3 

#Timeout (in seconds)  when connecting to an LDAP Server. The default value (and most reasonable) is 5 seconds.
ldap_timeout = 5

#Turn on or off the self-registration feature
self_registration = on

#The expiration time (in minute) of token created by token service, default is 30 minutes
token_expiration = 30

#The flag to control what users have permission to create projects
#The default value "everyone" allows everyone to creates a project. 
#Set to "adminonly" so that only admin user can create project.
project_creation_restriction = everyone

#Determine whether the job service should verify the ssl cert when it connects to a remote registry.
#Set this flag to off when the remote registry uses a self-signed or untrusted certificate.
verify_remote_cert = on
#************************END INITIAL PROPERTIES************************
#############
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105

创建配置文件目录，生成证书

mkdir -p /data/cert
cd /data/cert/

# 生成证书
openssl genrsa -des3 -out server.key 2048
openssl req -new -key server.key -out server.csr

# 备份证书
cp server.key server.key.org

# 由于docker在引导时是使用nginx当前端，如果启动时私钥或者证书有密码，就会失败，所以需要退出密码
openssl rsa -in server.key.org -out server.key

# 签名
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

# 授权证书
chmod a+x *
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

安装

cd /usr/local/harbor/
./install.sh
1
2

配置master、node1、node2、harbor的hosts文件，让他解析到harbor主机
192.168.66.205为lzharbor的主机电脑

echo "192.168.66.205 hub.atguigu.com" >> /etc/hosts
1

修改windows电脑的hosts文件
C:\Windows\System32\drivers\etc目录下的hosts文件可能无法修改，可以右键属性去除掉“只读”，然后使用notepad++进行修改，在最下面添加harborIp和域名，如图

访问harbor地址，这里哦

默认的账号是：admin，默认密码是：Harbor12345

尝试推送镜像到私有仓库

# 拉取一个镜像到本地
docker login https://hub.atguigu.com
# 打包拉取下来的镜像
docker tag wangyanglinux/myapp:v1 hub.atguigu.com/library/myapp:v1
# 推送到私有仓库
docker push hub.atguigu.com/library/myapp:v1
1
2
3
4
5
6

删除打包和上传的镜像，方便后面做测试

docker rmi -f hub.atguigu.com/library/myapp:v1
docker rmi -f wangyanglinux/myapp:v1
1
2

---

测试

删除错误的pod和deployment
由于docker run指令输入的有点问题，所以导致系统中有一个错误的pod存在，并且无法启动，所以打算先删除后再运行,没有错误的话可以跳过这段

查看pods，并删除

kubectl get pods
kubectl delete pod nginx-deployment-9fd7759fd-mnczr
1
2

查看deployment并删除

kubectl get deployment
kubectl delete deployment nginx-deployment
1
2

正式测试（在master节点上进行的测试）

创建一个副本数为1的deployment

kubectl run nginx-deployment --image=hub.atguigu.com/library/myapp:v1 --port=80 --replicas=1
1

查看情况

# 查看deployment
kubectl get deployment

# 查看链接的rs
kubectl get rs

# 查看pod以及对应的状态
kubectl get pod

# 查看pod更加详细的信息
kubectl get pod -o wide
1
2
3
4
5
6
7
8
9
10
11

查看对应的容器运行情况

ckubectl get pod -o wide查看对应的运行节点，在node1上运行

打开node1终端，查看docker的nginx进程是否存在，确实存在

docker ps -a  | grep nginx
1

访问，可以正常访问

curl 10.244.1.4/hostname.html
1

测试删除pod，删除后重新生成新的pod，由于启动时设置的replicas=1参数导致

kubectl get pod
kubectl delete pod nginx-deployment-85756b779-2p8jh
kubectl get pod
1
2
3

测试设置多个副本，删除副本后是否能保持设置的多个副本数量

设置多个副本

kubectl scale --replicas=3 deployment/nginx-deployment
1

查看副本数量并随机删除一个

kubectl get pod -o wide
kubectl delete pod nginx-deployment-85756b779-4mfbw
1
2

通过svc设置服务的暴露端口
将容器内部的80端口映射到svc的30000端口

kubectl expose deployment nginx-deployment --port=30000 --target-port=80
1

查看svc对应的ip

kubectl get svc 
1

访问

curl 10.108.208.182:30000
1

查看负载均衡规则

ipvsadm -Ln
1

kubectl get pod -o wide
1

发现一个问题，无法在master节点访问CLUSTER-IP，只能在自己的节点访问，所以在这里有一篇单独的帖子解决，可以参考一下：k8s排查无法访问服务

最后达到在浏览器中访问masterIP+NodePort映射出来的端口，能看到效果并且多次访问实现hostname改变，实现了负载均衡
的效果

大功告成，最后网络的那个问题卡了两天真的是心累啊，不过最后好的有点莫名其妙真的是糊了

余生还长，切勿惆怅