NSX ALB + Harbor + OpenShift 4.8 UPI安装配置实验笔记系列目录

目录

1 声明基础临时环境变量

2 为RHCOS的core用户准备ssh key

3 基于本地Registry的Image解压openshift-install安装程序

4 生成install-config.yaml文件

4.1 准备本地Registry证书

4.2 使用变量生成install-config.yaml文件

5 生成ignition文件

5.1 启用Operator主机的http服务

5.2 创建mainfest文件

5.3 按需修改配置文件

5.3.1 修改应用默认域名

5.3.2 配置NTP

5.4 生成Ignition文件

5.5 创建Ignition引导文件http下载目录

---

1 声明基础临时环境变量

因OpenShift会有版本更新情况，当非连续性操作时，可在此使用实际我们上面Download下来的RELEASE版本号更从新声明变量：

export OCP_RELEASE=4.8.36
export RHCOS_RELEASE=4.8.14
export LOCAL_REGISTRY='map.corp.tanzu'
export LOCAL_REPOSITORY='openshift/ocp4.8.36'
export PRODUCT_REPO='openshift-release-dev'
export RELEASE_NAME='ocp-release'
export ARCHITECTURE='x86_64'
export OCP_PATH=/data/OCP-${OCP_RELEASE}/ocp
export LOCAL_SECRET_JSON=${OCP_PATH}/secret/pull-secret.json
export REMOVABLE_MEDIA_PATH=${OCP_PATH}/ocp-image
export BOOT_FILE_PATH=/data/boot-files
export RHCOS_ISO_PATH=${BOOT_FILE_PATH}/rhcos-iso
export DOMAIN=corp.tanzu
export OCP_CLUSTER_ID=ocp
export OPERATOR_DOMAIN=operator.${DOMAIN}
export IGN_PATH=${BOOT_FILE_PATH}/ignition/${OCP_CLUSTER_ID}

2 为RHCOS的core用户准备ssh key

1). 在安装文件目录“/data/boot-files/ignition/ocp”建个ssh-key目录，用于存放我们生成的ssh key：

mkdir -p ${IGN_PATH}/ssh-key
ssh-keygen -t rsa -b 4096 -N '' -f ${IGN_PATH}/ssh-key/id_rsa
eval "$(ssh-agent -s)"
ssh-add ${IGN_PATH}/ssh-key/id_rsa

2). 为生成的ssh key声明环境变量：

export SSH_PRI_FILE=${IGN_PATH}/ssh-key/id_rsa 
export SSH_PUB_STR=$(cat ${IGN_PATH}/ssh-key/id_rsa.pub)

3 基于本地Registry的Image解压openshift-install安装程序

oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"
cp openshift-install /usr/sbin/
openshift-install version

4 生成install-config.yaml文件

4.1 准备本地Registry证书

将本地Registry的CA证书copy至准备目录：

cp /etc/pki/ca-trust/source/anchors/map-harbor.crt ${IGN_PATH}

4.2 使用变量生成install-config.yaml文件

1). 在Operator主机输入以下命令信息，此命令使用变量在yaml文件中同时更新了PullSecret、sshKey、本地Registry的ca证书、还有本地Registry的域名等信息：

cat << EOF > ${IGN_PATH}/install-config.yaml
apiVersion: v1
baseDomain: ${DOMAIN}
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ${OCP_CLUSTER_ID}
networking:
  clusterNetworks:
  - cidr: 100.224.0.0/16
    hostPrefix: 24
  networkType: OpenShiftSDN
  serviceNetwork:
  - 100.225.0.0/16
platform:
  none: {}
fips: false
pullSecret: '$(awk -v RS= '{$1=$1}1' ${LOCAL_SECRET_JSON})'
sshKey: '${SSH_PUB_STR}'
additionalTrustBundle: |
$(cat ${IGN_PATH}/map-harbor.crt | sed 's/^/  /g' | sed 's/^/  /g')
imageContentSources: 
- mirrors:
  - ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
EOF

2). 完成后使用以下命令查看一下yaml文件的内容：

more ${IGN_PATH}/install-config.yaml

3). 确认没有问题后，可以将其备份一份，因为生成ignition过程中此yaml会被销毁：

cp ${IGN_PATH}/install-config.yaml{,.`date '+%s'`.bak}

5 生成ignition文件

注：创建完ignition文件后，必须在24小时内完成OpenShift集群的创建，否则证书会过期失效。

5.1 启用Operator主机的http服务

yum -y install httpd
systemctl enable httpd
systemctl start httpd
systemctl status httpd

5.2 创建mainfest文件

1). 创建mainfest文件

openshift-install create manifests --dir ${IGN_PATH}

2). 查看成生成的结果

tree ${IGN_PATH}/manifests/ ${IGN_PATH}/openshift/

3). 修改Master节点不参与业务POD调度配置（可选 ）

sed -i 's/mastersSchedulable: true/mastersSchedulable: false/g' ${IGN_PATH}/manifests/cluster-scheduler-02-config.yml

4). 检查配置结果

cat ${IGN_PATH}/manifests/cluster-scheduler-02-config.yml

5.3 按需修改配置文件

5.3.1 修改应用默认域名

Openshift应用默认子域名为“apps”，如需要调，可在此时修改cluster-ingress-02-config.yml内的参数，此ymal文件内容如下：

修改命令如下，修改域名为dev.ocp.corp.tanzu：

sed -i 's/apps.ocp/dev.ocp/g' ${IGN_PATH}/manifests/cluster-ingress-02-config.yml

 

5.3.2 配置NTP

1). 分别为worker和master生成chrony.bu文件，具体如下：

Worker：

cat << EOF > 99-worker-chrony.bu
variant: openshift
version: 4.8.0
metadata:
  name: 99-worker-chrony 
  labels:
    machineconfiguration.openshift.io/role: worker 
storage:
  files:
  - path: /etc/chrony.conf
    mode: 0644 
    overwrite: true
    contents:
      inline: |
        pool 192.168.100.1 iburst 
        driftfile /var/lib/chrony/drift
        makestep 1.0 3
        rtcsync
        logdir /var/log/chrony
EOF

Master：

cat << EOF > 99-master-chrony.bu
variant: openshift
version: 4.8.0
metadata:
  name: 99-master-chrony 
  labels:
    machineconfiguration.openshift.io/role: master
storage:
  files:
  - path: /etc/chrony.conf
    mode: 0644 
    overwrite: true
    contents:
      inline: |
        pool 192.168.100.1 iburst 
        driftfile /var/lib/chrony/drift
        makestep 1.0 3
        rtcsync
        logdir /var/log/chrony
EOF

2). 生成对应的yaml文件，并存入安装路径下的openshift目录中：

butane 99-worker-chrony.bu -o ${IGN_PATH}/openshift/99-worker-chrony.yaml
butane 99-master-chrony.bu -o ${IGN_PATH}/openshift/99-master-chrony.yaml

3). 如果环境已安装，则可以用以下命令应用：

oc apply -f 99-worker-chrony.yaml
oc apply -f 99-master-chrony.yaml

5.4 生成Ignition文件

在我们安装目录中生成ignition文件并查看生成结果：

openshift-install create ignition-configs --dir ${IGN_PATH}/
ls -al ${IGN_PATH}/*.ign

5.5 创建Ignition引导文件http下载目录

1). 给安装文件目录配置权限：

chmod -R 755 ${IGN_PATH}
chmod -R 600 ${IGN_PATH}/ssh-key

注：如果ssh-key目录或文件权限过大，在ssh时会有“Permissions 0755 for '/data/boot-files/ignition/ocp/ssh-key/id_rsa' are too open.”报错提示。

2). 配置httpd config：

cat << EOF > /etc/httpd/conf.d/ignition.conf
Alias /ignition "${IGN_PATH}/../"
<Directory "${IGN_PATH}/../">
  Options +Indexes +FollowSymLinks
  Require all granted
</Directory>
<Location /ignition >
  SetHandler None
</Location>
EOF

3). 重启http服务：

systemctl restart httpd

4). 检查此http服务是否正常

curl http://${OPERATOR_DOMAIN}/ignition/ocp/