SpringBoot集成kubernetes-client升级k8s后初始化失败问题

SpringBoot集成kubernetes-client升级k8s后初始化失败问题

1.问题描述

程序以前使用的k8s版本是1.16，fabric8.kubernetes-client的版本是4.10.2，springboot版本是2.3.5。由于环境切换，这次需要升级k8s的版本，现在将k8s版本升级到了1.26，程序中使用了新配置的certificate-authority-data，client-certificate-data等。程序中原本初始化写法如下：

config = new ConfigBuilder().withMasterUrl(server)
        .withCaCertData(certificateAuthorityData)
        .withClientCertData(clientCertificateData)
        .withClientKeyData(clientKeyData)
        .build();
LogUtil.info(LogEnum.BIZ_K8S, "config信息为{}", JSON.toJSONString(config));
client = new DefaultKubernetesClient(config);
1
2
3
4
5
6
7

但是程序启动缺出现了问题，问题如下：

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.dubhe.k8s.utils.K8sUtils]: Factory method 'k8sUtils' threw exception; nested exception is io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.10.RELEASE.jar:5.2.10.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650) ~[spring-beans-5.2.10.RELEASE.jar:5.2.10.RELEASE]
	... 61 common frames omitted
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:234) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.BaseClient.<init>(BaseClient.java:51) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:133) ~[kubernetes-client-4.10.2.jar:na]
	at org.dubhe.k8s.utils.K8sUtils.<init>(K8sUtils.java:93) ~[classes/:na]
	at org.dubhe.k8s.config.K8sConfig.k8sUtils(K8sConfig.java:91) ~[classes/:na]
	at org.dubhe.k8s.config.K8sConfig$$EnhancerBySpringCGLIB$$77270501.CGLIB$k8sUtils$10(<generated>) ~[classes/:na]
	at org.dubhe.k8s.config.K8sConfig$$EnhancerBySpringCGLIB$$77270501$$FastClassBySpringCGLIB$$1f42aed6.invoke(<generated>) ~[classes/:na]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.10.RELEASE.jar:5.2.10.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.10.RELEASE.jar:5.2.10.RELEASE]
	at org.dubhe.k8s.config.K8sConfig$$EnhancerBySpringCGLIB$$77270501.k8sUtils(<generated>) ~[classes/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_321]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_321]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_321]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_321]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.10.RELEASE.jar:5.2.10.RELEASE]
	... 62 common frames omitted
Caused by: java.io.IOException: Invalid DER: object is not integer
	at io.fabric8.kubernetes.client.internal.PKCS1Util$Asn1Object.getInteger(PKCS1Util.java:125) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.PKCS1Util.next(PKCS1Util.java:55) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.PKCS1Util.decodePKCS1(PKCS1Util.java:46) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.CertUtils.handleOtherKeys(CertUtils.java:175) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.CertUtils.loadKey(CertUtils.java:135) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:113) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:245) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122) ~[kubernetes-client-4.10.2.jar:na]
	at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:120) ~[kubernetes-client-4.10.2.jar:na]
	... 77 common frames omitted
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

2.问题分析

然后就上网去查找资料，但是搜索了很多，都没有找到有用的信息。只好自己一点点debug去查看源代码。debug的时候看到build的时候会先去读配置文件，这让我想到了之前做的一个项目，那个项目使用的是io.kubernetes client-java这个依赖包。我再开发的时候会将config文件放在系统路径下。又换了一种创建方式：

Config config = new ConfigBuilder().build();
1

于是按照这种方式也将config放在相应的路径下，程序果然可以去建立连接，自测创建pod也是成功的。

由于一个读取的是配置文件，一个读取的是配置在nacos配置中心的配置，两个的配置信息毫无差别，但是执行结果就是不一样。然后再去debug去看两中配置问题到底有何不一样。最后发现Config中的clientKeyAlgo中赋值不同，读取nacos中的配置产生的算法是RSA，读取config配置文件产生的是EC。

然后我将系统路径下的config文件移除，将原本程序改为：

Config config = new ConfigBuilder().withMasterUrl(server)         .withCaCertData(certificateAuthorityData)         .withClientCertData(clientCertificateData)         .withClientKeyData(clientKeyData)         .withClientKeyAlgo("EC")         .build();
1

发现也成功了。由于时间关系，就没有仔细去看为什么读取nacos中的配置产生的是RSA，读取config配置文件产生的是EC。

3.解决办法

最终解决方案是：

1.在程序启动的包设置config环境变量，让程序读取config文件；设置了config文件后，就可以用Config config = new ConfigBuilder().build();这种方式创建。

2.依然去读取nacos中的配置（certificate-authority-data，client-certificate-data），在程序Builder时手动设置ClientKeyAlgo。