kube-apiserver 安装路由汇总

本文主要总结下kube-apiserver的三个链式server安装的路由信息，并举例说明收到http请求后的处理流程。

路由信息

aggregator-server
对于aggregator-server来说，它主要安装了用于注册apiservice的路由apiregistration.k8s.io信息到goRestfulContainer，每当收到apiservice请求时，就把相应的信息安装到NonGoRestfulMux，比如kube-apiserver进程启动时，就会把apiserver和extensions-server的所有资源，包括进程启动后通过crd创建的资源，自动识别并注册到apiservice中，所以说aggregator-server是一个聚合器，收到http请求后，先经过aggregator-server处理，根据请求的path转发到不同的server进一步处理。如果aggregator-server没找到path对应的handler，也要通过not-found转给下一个server处理。

kube-apiserver
kube-apiserver主要安装restful路由信息，包括/api/下的核心资源和/apis/下的分组资源，对于处理不了的请求也会转发给extensions-server。

extensions-server
extensions-server主要安装了处理crd的路由。到达这里的请求，如果请求path是/apis/apiextensions.k8s.io/v1/customresourcedefinitions
则为创建自定义资源，如果不是则为对自定义资源的操作。对于处理不了的请求，返回404给客户端。

http请求举例

下面举几个例子，看一下收到不同的http请求时，分别经过哪些流程。这里通过分析log的方式进行验证，使用kubectl工具模拟客户端，再执行时加上参数 -v 8查看详细信息，对于服务端修改apiserver的配置文件/etc/kubernetes/mainfests/kube-apiserver.yaml，添加参数–v=5可从查看到详细log。
获取api-versions时
从下面log可看到当执行api-versions时，主要发送了两个http请求: https://192.168.247.129:6443/api和https://192.168.247.129:6443/apis

root@test:~# kubectl api-versions -v 8
I0814 04:14:24.336136   27841 loader.go:372] Config loaded from file:  /root/.kube/config
I0814 04:14:24.362015   27841 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191" 'https://192.168.247.129:6443/api?timeout=32s'
...
I0814 04:14:24.402531   27841 request.go:1181] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.247.129:6443"}]}
I0814 04:14:24.407561   27841 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191" 'https://192.168.247.129:6443/apis?timeout=32s'
...
I0814 04:14:24.433150   27841 request.go:1181] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},
...
[{"groupVersion":"crd.projectcalico.org/v1","version":"v1"}],"preferredVersion":{"groupVersion":"crd.projectcalico.org/v1","version":"v1"}}]}
...
1
2
3
4
5
6
7
8
9
10
11

kube-apiserver对请求处理的log如下：
对/api 的get请求匹配到安装到kube-aggregator NonGoRestfulMux中的精确路由/api，此路由为local类型的，所以直接调用到kube-apiserver。在kube-apiserver中匹配到安装到goRestfulContainer中的精确路由/api，执行其对应的handler，返回相应信息。

I0814 05:06:17.104452       1 handler.go:153] kube-aggregator: GET "/api" satisfied by nonGoRestful
I0814 05:06:17.104505       1 pathrecorder.go:240] kube-aggregator: "/api" satisfied by exact match
I0814 05:06:17.104525       1 handler.go:143] kube-apiserver: GET "/api" satisfied by gorestful with webservice /api
I0814 05:06:17.104850       1 httplog.go:104] "HTTP" verb="GET" URI="/api?timeout=32s" latency="1.92806ms" userAgent="kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191" audit-ID="e02ee0c5-54eb-4751-a537-288361ce5a7a" srcIP="192.168.247.129:52554" resp=200
1
2
3
4

对 /apis 的get请求匹配到安装到kube-aggregator NonGoRestfulMux中的精确路由/apis，并执行对应的apisHandler，返回所有的组信息，并没有将请求转给其他server。因为kube-aggregator以apiservice的方式聚合了kube-apiserver和kube-extensions的所有路由，直接返回信息即可，没必要再转发给其他server。

I0814 05:06:17.138417       1 handler.go:153] kube-aggregator: GET "/apis" satisfied by nonGoRestful
I0814 05:06:17.138471       1 pathrecorder.go:240] kube-aggregator: "/apis" satisfied by exact match
I0814 05:06:17.139457       1 httplog.go:104] "HTTP" verb="GET" URI="/apis?timeout=32s" latency="1.699395ms" userAgent="kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191" audit-ID="eedd5a82-dedb-4689-b28c-1d8b1feaa7f5" srcIP="192.168.247.129:52554" resp=200
1
2
3

获取pod信息时
当获取pod信息时，发送的http请求为https://192.168.247.129:6443/api/v1/namespaces/default/pods，kube-apiserver对请求处理的log如下，可看到首先匹配到安装到kube-aggregator NonGoRestfulMux中的前缀路由/api/，此路由为local类型的，所以直接调用到kube-apiserver。在kube-apiserver中匹配到安装到goRestfulContainer中的精确路由/api/v1/namespaces/default/pods，执行其对应的handler，返回相应信息。

I0919 15:59:21.133667       1 handler.go:153] kube-aggregator: GET "/api/v1/namespaces/default/pods" satisfied by nonGoRestful
I0919 15:59:21.133765       1 pathrecorder.go:247] kube-aggregator: "/api/v1/namespaces/default/pods" satisfied by prefix /api/
I0919 15:59:21.133796       1 handler.go:143] kube-apiserver: GET "/api/v1/namespaces/default/pods" satisfied by gorestful with webservice /api/v1
I0919 15:59:21.141189       1 httplog.go:104] "HTTP" verb="LIST" URI="/api/v1/namespaces/default/pods?limit=500" latency="9.213611ms" userAgent="kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191" audit-ID="8344c169-005e-4291-8fcc-355fd2369bcd" srcIP="192.168.247.129:48746" resp=200
1
2
3
4

获取cr信息时
这里以calico cni创建的crd为例说明，发送的http请求为https://192.168.247.129:6443/apis/crd.projectcalico.org/v1，kube-apiserver对请求处理的log如下，可看到首先匹配到安装到kube-aggregator NonGoRestfulMux中的精确路由/apis/crd.projectcalico.org/v1，此路由为local类型的，所以直接调用到kube-apiserver。在kube-apiserver匹配不到任何路由，通过NotFoundHandler转给apiextensions-apiserver，在这里匹配到安装到nonGoRestful的前缀路由/apis/，执行其handler，从etcd获取信息后返回给客户端。

I0919 15:52:16.653662       1 queueset.go:375] QS(workload-low): Dispatching request &request.RequestInfo{IsResourceRequest:false, Path:"/apis/crd.projectcalico.org/v1", Verb:"get", APIPrefix:"apis", APIGroup:"", APIVersion:"", Namespace:"", Resource:"", Subresource:"", Name:"", Parts:[]string(nil)} &user.DefaultInfo{Name:"system:serviceaccount:default:default", UID:"c70432d8-95a7-4f69-bb3f-f87cf30bad65", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:default", "system:authenticated"}, Extra:map[string][]string(nil)} from its queue
I0919 15:52:16.654350       1 handler.go:153] kube-aggregator: GET "/apis/crd.projectcalico.org/v1" satisfied by nonGoRestful
I0919 15:52:16.654416       1 pathrecorder.go:240] kube-aggregator: "/apis/crd.projectcalico.org/v1" satisfied by exact match
I0919 15:52:16.654456       1 handler.go:153] kube-apiserver: GET "/apis/crd.projectcalico.org/v1" satisfied by nonGoRestful
I0919 15:52:16.654513       1 pathrecorder.go:253] kube-apiserver: "/apis/crd.projectcalico.org/v1" satisfied by NotFoundHandler
I0919 15:52:16.654536       1 handler.go:153] apiextensions-apiserver: GET "/apis/crd.projectcalico.org/v1" satisfied by nonGoRestful
I0919 15:52:16.654555       1 pathrecorder.go:247] apiextensions-apiserver: "/apis/crd.projectcalico.org/v1" satisfied by prefix /apis/
I0919 15:52:16.659291       1 httplog.go:104] "HTTP" verb="GET" URI="/apis/crd.projectcalico.org/v1" latency="33.802702ms" userAgent="curl/7.68.0" audit-ID="400cfe3a-c487-4a21-af7c-6733892fb219" srcIP="192.168.247.129:45016" resp=200
1
2
3
4
5
6
7
8