Kubesphere中DevOps流水线无法部署/部署失败

摘要

总算能让devops运行以后，流水线却卡在了deploy这一步。碰到了两个比较大的问题，一个是无法使用k8sp自带的kubeconfig认证去部署；一个是部署好了以后但是没有办法解析镜像名。

版本信息

k8s：v1.21.5
k8sp：v3.3.0

流水线概览

Q问题描述

pipeline 在deploy 的阶段总是报各种错。

Q1.使用k8sp自带kube认证产生报错

      stage('deploy fail') {
        agent none
        steps {
          withCredentials([kubeconfigContent(credentialsId : 'kubeconfigger' ,variable : 'KUBECONFIGGER' ,)]) {
            kubernetesDeploy(enableConfigSubstitution: true, deleteResource: false, kubeconfigId: 'kubeconfigger', configs: 'hospital-manage/deploy/**')
          }

        }
      }

报错内容如下：

Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/redp5lk5/rose/hospital-manage/deploy/deploy.yml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: 
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: 
    at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:213)
    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:201)
    at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
    at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
    at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
    at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:376)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
    at java.base/java.lang.Thread.run(Unknown Source)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.81.183/10.233.81.183:49480
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1001)
        at hudson.FilePath.act(FilePath.java:1256)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
        at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
        at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
        at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
        at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
        at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
        at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
        at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: 
    at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
    at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
    at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeploymentWithHttpInfo(AppsV1Api.java:7299)
    at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeployment(AppsV1Api.java:7275)
    at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:210)
    ... 16 more
Api call failed with code 400, detailed message: {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "the export parameter, deprecated since v1.14, is no longer supported",
  "reason": "BadRequest",
  "code": 400
}
Kubernetes deployment ended with HasError

从message来看，已经不支持这个什么什么参数了。从与小伙伴的沟通看，以前的版本是可以用的。但是新版本不支持了。

解决方案

采用以下写法。
缺点：不支持图形化流水线编辑，点击编辑该凭证会闪退。

法1

需要在环境变量处声明 KUBECONFIG_CREDENTIAL_ID

    stage('deploy success') {
      agent none
      steps {
        container('maven') {
          withCredentials([kubeconfigFile(
                                           credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) {
              sh 'envsubst < hospital-manage/deploy/deploy.yml | kubectl apply -f -'
            }

          }

        }
      }

法2

与法1没什么区别。主要区别就是shell中的命令。

    stage('deploy hospital-manage to dev') {
      agent none
      steps {
        container('maven') {
          withCredentials([kubeconfigFile(credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')]) {
            sh 'kubectl apply -f hospital-manage/deploy/**'
          }

        }

      }
    }

Q2：无法解析镜像名

在deploy.yml文件中，有关镜像名的描述如下所示：

spec:
  containers:
    - image: $DOCKERHUB_NAMESPACE/server-gateway:SNAPSHOT-$BUILD_NUMBER

但是却出现了如下结果

状态信息
初始化完成
状态：True
容器组就绪
状态：False
原因：ContainersNotReady
消息：containers with unready status: [app]
所有容器就绪
状态：False
原因：ContainersNotReady
消息：containers with unready status: [app]
容器组调度完成
状态：True
无法解析镜像名称

无法解析镜像名，我点进去这个pod的yml以后，才发现，流水线没能解析出这些变量。才导致了无法解析镜像名。

解决方案

治标不治本方案

直接把镜像写死

belchance/ruoyi:hospital-manage_SNAPSHOT-10

替换环境变量方案

https://kubesphere.io/forum/d/5309-deploy-to-kubernets
来源于社区网友，方法就是使用 envsubst，把环境yml文件里的环境变量改了。
注意替换的字符要在JenkinsFile的环境变量区声明，deploy.yml的位置要准确。

  agent none
  steps {
    container('nodejs') {
      sh 'envsubst \\'${REGISTRY},${ALIYUNHUB_NAMESPACE},${BUILD_NUMBER}\\' < deploy/deploy.yml > deploy/deploy2.yml'
      sh 'cat deploy/deploy2.yml'
    }
  }
}

envsubst用法介绍：

envsubst '$DOCKERHUB_NAME,$NUMBER' < deploy.yml 
envsubst '需要替换的环境变量' < target.file
command < file     将输入重定向到 file。