多数据中心集群,某个集群所有节点都报这个异常。
- 2023-10-09T10:47:50.645+0800 [DEBUG] agent.router.manager: cycled away from server: server="08fb61b026134b4489b47721c2740d43 (Addr: tcp/10.70.51.58:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:50.645+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.70.51.58:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:50.646+0800 [DEBUG] agent.router.manager: cycled away from server: server="749d1fca243d44dc9c55452054a63c91 (Addr: tcp/10.10.6.64:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:50.646+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.10.6.64:50011 error="rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:50.648+0800 [DEBUG] agent.router.manager: cycled away from server: server="824e7c88248d464cbfc33047ed660f58 (Addr: tcp/10.10.6.81:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:50.648+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.10.6.81:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:54.435+0800 [DEBUG] agent.router.manager: cycled away from server: server="08fb61b026134b4489b47721c2740d43 (Addr: tcp/10.70.51.58:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:54.435+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.70.51.58:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.065+0800 [DEBUG] agent.router.manager: cycled away from server: server="749d1fca243d44dc9c55452054a63c91 (Addr: tcp/10.10.6.64:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:55.065+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.10.6.64:50011 error="rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.095+0800 [DEBUG] agent.router.manager: cycled away from server: server="824e7c88248d464cbfc33047ed660f58 (Addr: tcp/10.10.6.81:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:55.095+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.10.6.81:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.095+0800 [WARN] agent.cache: handling error in Cache.Notify: cache-type=connect-ca-root error="rpc error making call: rpc error making call: CA has not finished initializing" index=0
- 2023-10-09T10:47:55.095+0800 [TRACE] agent.proxycfg: Data source returned; handling snapshot update: kind=ingress-gateway proxy=5900d086ad684569961398306b5fdb83 service_id=5900d086ad684569961398306b5fdb83 correlationID=roots
- 2023-10-09T10:47:55.095+0800 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=5900d086ad684569961398306b5fdb83 service_id=5900d086ad684569961398306b5fdb83 id=roots error="error filling agent cache: rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.098+0800 [DEBUG] agent.router.manager: cycled away from server: server="08fb61b026134b4489b47721c2740d43 (Addr: tcp/10.70.51.58:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:55.098+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.70.51.58:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.099+0800 [DEBUG] agent.router.manager: cycled away from server: server="749d1fca243d44dc9c55452054a63c91 (Addr: tcp/10.10.6.64:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:55.099+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.10.6.64:50011 error="rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.102+0800 [DEBUG] agent.router.manager: cycled away from server: server="824e7c88248d464cbfc33047ed660f58 (Addr: tcp/10.10.6.81:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:55.102+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.10.6.81:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
- 2023-10-09T10:47:55.277+0800 [WARN] agent: Check socket connection failed: check=service:5900d086ad684569961398306b5fdb83 error="dial tcp 127.0.0.1:52001: connect: connection refused"
- 2023-10-09T10:47:55.277+0800 [WARN] agent: Check is now critical: check=service:5900d086ad684569961398306b5fdb83
- 2023-10-09T10:47:57.279+0800 [DEBUG] agent.router.manager: cycled away from server: server="08fb61b026134b4489b47721c2740d43 (Addr: tcp/10.70.51.58:50011) (DC: 88758884f12d4c6483f1193a59a185c8)"
- 2023-10-09T10:47:57.279+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Roots server=10.70.51.58:50011 error="rpc error making call: rpc error making call: CA has not finished initializing"
这个集群是个子集群,ca不归它管,只要重启这个集群所有节点就恢复正常了