[笔记] 记录docker-compose使用和Harbor的部署过程

容器技术

---

记录docker-compose使用和Harbor的部署过程

1. 使用CentOS 7.9系列的Linux操作系统
2. docker社区版：docker-ce
3. docker部署编排工具：docker-compose
4. docker私有镜像仓库管理工具：docker-harbor
5. docker轻量级可视化容器管理工具：Portainer

# 默认已初始化安装 CentOS 7.9 + docker-ce
CPU: 4核 * 2
Memory: 16G
Disk: 2块物理硬盘(sda,sdb) sda: 40GB(预装最小化Linux), sdb: 200GB
Swap: 12G
hostname: docker01.myside.com
ip: 10.0.0.210
gateway: 10.0.0.254
dns: 223.5.5.5 114.114.114.114
docker应用的映射存储目录: /opt/mydocker

设置docker服务端的目录结构

/opt/mydocker/
├── certs			# 存放证书的目录, 按项目名称划分, ca证书放当前目录下
│   ├── ca.crt		# CA根证书
│   ├── ca.key		# CA私钥
│   └── harbor		# Harbor项目的SSL证书目录
├── docker-root		# docker-ce应用的目录
├── packages		# 存储安装包、软件包
│   ├── harbor
│   └── harbor-offline-installer-v2.10.2.tgz
└── projects
    └── harbor		# Harbor项目的数据目录

Harbor

https方式部署：测试环境部署使用自签名SSL证书

测试环境部署
## harbor配置HTTPS方式部署, 测试环境使用自签名SSL证书, 正式环境使用企业颁发的SSL证书
# 配置CA证书(CN=mysite.com) 和 自签名SSL证书(CN=harbor.mysite.com SAN=DNS:harbor.mysite.com,DNS:*.harbor.mysite.com)
cd /opt/mydocker/certs
# 生成CA的私钥CA证书 ca.crt
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=mysite.com" \
 -key ca.key -out ca.crt
 
# 生成私钥和证书签名请求文件(CSR)
openssl genrsa -out harbor/harbor.mysite.com.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=harbor.mysite.com" \
    -key harbor/harbor.mysite.com.key -out harbor/harbor.mysite.com.csr

# 给harbor服务生成临时的x509 v3 扩展文件，使CA签发时附加SAN及更多扩展属性
cat <<EOF > v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.mysite.com
DNS.2=*.harbor.mysite.com
EOF
# 使用CA证书给CSR文件签发CRT证书
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in harbor/harbor.mysite.com.csr -out harbor/harbor.mysite.com.crt

https方式部署：正式环境部署使用企业颁发的SSL证书

正式环境部署
# 正式环境部署Harbor的SSL证书仅需要生成 私钥、证书签名请求(CSR)，使用企业CA给csr颁发crt证书即可
# 证书签名请求(CSR)一般要手工添加SAN扩展属性
openssl genrsa -out harbor/harbor.mysite.com.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=Technology Co., Ltd./OU=Department/CN=harbor.mysite.com" \
    -reqexts SAN -config <(cat /etc/pki/tls/openssl.cnf \
    <(printf "[SAN]\nsubjectAltName=DNS:harbor.mysite.com,*.harbor.mysite.com")) \
    -key harbor/harbor.mysite.com.key -out harbor/harbor.mysite.com.csr

给Docker守护进程添加Harbor的SSL证书认证

# Docker 守护进程会把 .crt当成CA证书 .cert当成客户端证书。所以这里要将harbor的ssl证书转换成后缀.cert的pem格式证书
# 文件内容是一样的可以用 cp ，不是pem格式则使用 openssl x509 -inform PEM 转换成pem格式
\cp harbor/harbor.mysite.com.crt harbor/harbor.mysite.com.cert
openssl x509 -inform PEM -in harbor/harbor.mysite.com.crt -out harbor/harbor.mysite.com.cert

# 复制证书文件到docker目录, 为后续的 docker login 提供证书认证。 login缺失下列文件将报错，见下文
mkdir -p /etc/docker/certs.d/harbor.mysite.com
cp ca.crt harbor/harbor.mysite.com.cert harbor/harbor.mysite.com.key /etc/docker/certs.d/harbor.mysite.com/

安装Harbor

安装 Harbor

## 官方tag包独立安装。  如下方法是获取到latest节点的版本，但仓库会存在有-rc的候选测试版本，建议从github仓库对照过最新的正式版本tag再构建二进制包的下载命令
HARBOR_TAG=`curl -s https://api.github.com/repos/goharbor/harbor/releases/latest | grep -i tag_name | awk -F '"' '{print $4}'`
wget -P /opt/mydocker/packages/ https://github.com/goharbor/harbor/releases/download/$HARBOR_TAG/harbor-offline-installer-$HARBOR_TAG.tgz

cd /opt/mydocker/packages/
tar zxvf harbor-offline-installer-$HARBOR_TAG.tgz; cd harbor
cp harbor.yml.tmpl harbor.yml
sed -i '/^$\|^\s*#/d' harbor.yml  	# 删除空行和注释行

# (可选)给core组件设置quota_update_provider，配额更新提供程序为redis
cat <<EOF >> harbor.yml
core:
  quota_update_provider: redis
  
EOF

# harbor.yml
hostname: harbor.mysite.com  # 主机名
http:
  port: 80
https:
  port: 443
  certificate: /opt/mydocker/certs/harbor/harbor.mysite.com.crt  # harbor的ssl证书，CN要跟hostname一致，否则会出现HSTS问题
  private_key: /opt/mydocker/certs/harbor/harbor.mysite.com.key  # harbor的ssl证书私钥
harbor_admin_password: Harbor12345
database:
  password: root123
  max_idle_conns: 100
  max_open_conns: 900
  conn_max_lifetime: 5m
  conn_max_idle_time: 0
data_volume: /opt/mydocker/projects/harbor	# 指定宿主机的数据目录
trivy:
  ignore_unfixed: false
  skip_update: false
  skip_java_db_update: false
  offline_scan: false
  security_check: vuln
  insecure: false
jobservice:
  max_job_workers: 10
  job_loggers:
    - STD_OUTPUT
    - FILE
  logger_sweeper_duration: 1 #days
notification:
  webhook_job_max_retry: 3
  webhook_job_http_client_timeout: 3 #seconds
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
_version: 2.10.0
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy
upload_purging:
  enabled: true
  age: 168h
  interval: 24h
  dryrun: false
cache:
  enabled: true  # 开启cache缓存
  expire_hours: 24
core:
  quota_update_provider: redis

[root@docker01 harbor]# ./install.sh  	# 执行harbor目录下的install.sh安装脚本
[Step 5]: starting Harbor ...
WARN[0000] /opt/mydocker/packages/harbor/docker-compose.yml: `version` is obsolete 
[+] Running 10/10
 ✔ Network harbor_harbor        Created
 ✔ Container harbor-log         Started 
 ✔ Container harbor-db          Started 
 ✔ Container harbor-portal      Started 
 ✔ Container registry           Started 
 ✔ Container registryctl        Started 
 ✔ Container redis              Started 
 ✔ Container harbor-core        Started 
 ✔ Container harbor-jobservice  Started 
 ✔ Container nginx              Started 
✔ ----Harbor has been installed and started successfully.----
# 出现如上字样 说明harbor安装完成了

验证Harbor

# 查看harbor的监听端口和配置
# 在/opt/mydocker/packages/harbor/ 目录下执行 docker compose [OPTIONS]
# 或任意目录下执行 docker compose -f /opt/mydocker/packages/harbor/docker-compose.yml [OPTIONS]
[root@docker01 harbor]# docker compose ps
WARN[0000] /opt/mydocker/packages/harbor/docker-compose.yml: `version` is obsolete 
NAME                IMAGE                                 COMMAND                  SERVICE       CREATED         STATUS                   PORTS
harbor-core         goharbor/harbor-core:v2.10.2          "/harbor/entrypoint.…"   core          6 minutes ago   Up 6 minutes (healthy)   
harbor-db           goharbor/harbor-db:v2.10.2            "/docker-entrypoint.…"   postgresql    6 minutes ago   Up 6 minutes (healthy)   
harbor-jobservice   goharbor/harbor-jobservice:v2.10.2    "/harbor/entrypoint.…"   jobservice    6 minutes ago   Up 6 minutes (healthy)   
harbor-log          goharbor/harbor-log:v2.10.2           "/bin/sh -c /usr/loc…"   log           6 minutes ago   Up 6 minutes (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       goharbor/harbor-portal:v2.10.2        "nginx -g 'daemon of…"   portal        6 minutes ago   Up 6 minutes (healthy)   
nginx               goharbor/nginx-photon:v2.10.2         "nginx -g 'daemon of…"   proxy         6 minutes ago   Up 6 minutes (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis               goharbor/redis-photon:v2.10.2         "redis-server /etc/r…"   redis         6 minutes ago   Up 6 minutes (healthy)   
registry            goharbor/registry-photon:v2.10.2      "/home/harbor/entryp…"   registry      6 minutes ago   Up 6 minutes (healthy)   
registryctl         goharbor/harbor-registryctl:v2.10.2   "/home/harbor/start.…"   registryctl   6 minutes ago   Up 6 minutes (healthy)   

[root@docker01 harbor]# iptables -t nat -nL DOCKER
[root@docker01 harbor]# netstat -tnlp | grep docker

构建自定义镜像

Dockerfile构建自定义镜像

# python+flask项目镜像, name:dingtalk-monitorbook	tag:1.0
# 自定义镜像的ENV环境变量建议以 _ 开头，方便区分基础镜像与自定义镜像的ENV参数
[root@docker01 dingtalk-monitorbook]# vim Dockerfile
# 一阶段构建
FROM python:3.9-buster as builder-image

# 安装py项目的requirements依赖文件
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple


# 二阶段构建
FROM python:3.9-slim-buster
# 作者信息
LABEL maintainer "ww1372247148@163.com"

# 复制python的二进制执行文件和包文件
COPY --from=builder-image /usr/local/bin /usr/local/bin
COPY --from=builder-image /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages

# 设置环境变量
ENV _CODE_DIR=/app \
    _APP_PORT=8000

# 推荐容器内使用非root用户运行进程
RUN groupadd -r _users; useradd -r -g _users _user
USER _user

# 设置工作目录
WORKDIR $_CODE_DIR
# 复制源码文件
COPY . .
# 暴露TCP端口给容器做端口映射
EXPOSE $_APP_PORT

# 添加网站的健康检查
HEALTHCHECK CMD curl --fail http://localhost:$_APP_PORT || exit 1

# 运行python命令
CMD ["python3","app.py"]

# 项目的main入口文件可能不是app.py，为保留项目的目录结构，使用 ln -s 挂载 app.py
[root@docker01 dingtalk-monitorbook]# ln -s [项目实际入口文件] ./app.py

# 新增.dockerignore文件, 根据项目目录结构添加忽略的文件
[root@docker01 dingtalk-monitorbook]# vim .dockerignore
Dockerfile
README.md
.git/

# Dockerfile构建镜像
docker build -t wwzhg77777/dingtalk-monitorbook:1.0 -f Dockerfile .
# 查看镜像构建历史
docker history wwzhg77777/dingtalk-monitorbook:1.0

docker run运行镜像，生成容器

# docker run 测试镜像是否可运行. 推荐设置 限制CPU核数、内存容量硬限制、容器不使用swap内存
docker run -itd --name dingtalk-monitorbook -p 8000:8000 --restart=unless-stopped --cpus=1 -m 256m --memory-swappiness=0  wwzhg77777/dingtalk-monitorbook:1.0
# 查看容器运行日志
docker logs dingtalk-monitorbook

docker compose up -d运行容器编排，生成容器

# docker-compose.yml
version: "3"
services:
  dingtalk-monitorbook:
    container_name: dingtalk-monitorbook
    ports:
      - 8000:8000
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: "1"
          memory: 256m
    networks:
      - dingtalk-monitorbook
    image: wwzhg77777/dingtalk-monitorbook:1.0
networks:
  dingtalk-monitorbook:
    name: dingtalk-monitorbook

# docker-compose 测试镜像是否可运行
docker compose up -d
# 查看容器运行日志
docker compose logs

(可选)使用composerize生成docker-compose.yml

# (可选)使用 Composerize 将Container running之后的容器配置转换成 docker-compose.yml 再进行优化
# Composerize github: https://github.com/composerize/composerize
# Composerize的在线网站: https://www.composerize.com/
# Composerize的容器下载(仅支持web访问): https://hub.docker.com/r/gettionhub/composerize
# 自己构建一个composerize镜像, 设置alias别名 用于CLI执行composerize
cat <<EOF > Dockerfile
FROM node:14-alpine
RUN npm install composerize -g
ENTRYPOINT ["composerize"]
EOF
docker build -t local-composerize .
echo "alias composerize='docker run -it --rm local-composerize'" >> /etc/profile.d/_alias.sh
source /etc/profile.d/_alias.sh
[root@docker01 ~]# composerize docker run -itd --name dingtalk-monitorbook -p 8000:8000 --restart=unless-stopped --cpus=1 -m 256m wwzhg77777/dingtalk-monitorbook:1.0
name: <your project name>
services:
  dingtalk-monitorbook:
    stdin_open: true
    tty: true
    container_name: dingtalk-monitorbook
    ports:
      - 8000:8000
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: 1
          memory: 256m
    image: wwzhg77777/dingtalk-monitorbook:1.0

上传镜像到harbor，修改harbor的镜像标签

# Docker守护进程已添加Harbor的SSL证书认证，执行 docker login 登录
[root@docker01 ~]# docker login -u admin -p Harbor12345 harbor.mysite.com
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
# 出现如上字样 说明docker登录harbor成功

# 给本地的镜像打tag，加上harbor服务器地址作为前缀，默认放library项目下
docker tag wwzhg77777/dingtalk-monitorbook:1.0 harbor.mysite.com/library/dingtalk-monitorbook:1.0
# push镜像到harbor
docker push harbor.mysite.com/library/dingtalk-monitorbook:1.0

Harbor注意事项

### Docker守护进程没有添加Harbor的SSL证书认证，测试登录Harbor
[root@docker02 ~]# docker login -u admin -p Harbor12345 harbor.mysite.com
Error response from daemon: Get "https://harbor.mysite.com/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority
# 添加CA证书到 /etc/docker/certs.d/harbor.mysite.com/ 目录即可解决

### push镜像时, 如果不加harbor服务器地址作为前缀, 将默认push到docker.io
[root@docker01 ~]# docker push wwzhg77777/dingtalk-monitorbook:1.0
The push refers to repository [docker.io/wwzhg77777/dingtalk-monitorbook]

参考来源

1. Harbor Installation and Configuration
2. python 多阶段构建docker镜像，有效减少镜像大小
3. 24条Dockerfile及指令最佳实践
4. 如何编写最佳的Dockerfile
5. Docker 8：Docker 资源（内存/CPU）限制实验