• 【博客528】k8s serviceTopo原理

    k8s serviceTopo原理

    serviceTopo使用场景：

    流量亲和性绑定：

    有一个daemonset服务，其对应一个service，我们希望在node上的服务通过service访问时，
    能够优先命中本node上的daemonset后端pod，当本node不存在时才访问其它node的daemonset pod

    serviceTopo example:

    1、希望流量优先转发到本node的pod，如果不存在就转发到其他node的pod

    apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx
  topologyKeys:
    - "kubernetes.io/hostname"
    - "*"

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15

    2、希望流量优先转发到本node的pod，如果不存在就转发到其他node的pod

    apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx
  topologyKeys:
    - "kubernetes.io/hostname"

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14

    针对两种情况，探究一下kube-proxy如何来实现serviceTopo

    1、希望流量优先转发到本node的pod，如果不存在也不转发到其他node的pod

    部署：

    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx
  topologyKeys:
    - "kubernetes.io/hostname"

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33

    查看部署结果：

    [root@10-231-2-19 ~]# k get pod -o wide
NAME                                  READY   STATUS        RESTARTS   AGE   IP            NODE              NOMINATED NODE   READINESS GATES
nginx-deployment-j5wfx                1/1     Running       0          63m   10.244.2.51   10.231.2.21                  
nginx-deployment-pdhhz                1/1     Running       0          63m   10.244.1.45   10.231.2.20                  
nginx-deployment-x2fgf                1/1     Running       0          63m   10.244.5.70   10.231.2.46                  

[root@10-231-2-19 ~]# k get service
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes                    ClusterIP   10.96.0.1               443/TCP        99d
nginx                         ClusterIP   10.102.57.42            80/TCP         63m

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10

    查看带有nginx pod的node上的nginx service iptables规则：只设置了本node的pod转发

    我们查看了 10.231.2.20上的规则，其设置dnat规则刚好是本node上的10.244.1.45这个pod

    -A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx" -j KUBE-SEP-YTKAPZ6VEZPKTH4H
-A KUBE-SEP-YTKAPZ6VEZPKTH4H -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.244.1.45:80

    • 1
    • 2

    查看没有nginx pod的node上的nginx service iptables规则：没有对应规则，流量丢弃

    // 可以看到此时没有nginx service对应的iptables链
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-6E3L2NSY6W6DVRUL -m comment --comment "default/test-iptables" -j KUBE-SEP-GTKVEF4YNPXHWOH2
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-T62R2ZCQJ7L2XKT3
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-36PH6GXFEN4T4CYJ
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-TGXK4KBOH5HJRS4R
-A KUBE-SVC-JD5MR3NA4I4DYORP -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-SEP-7W3ALXIW4DWCNY6Q
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-TJFI7B7HUN7S5YVZ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-KI54576H4CGHAD2A
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LSIUSH75GR7U3TUR
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-CWXUMBMH75C7ZUMU
COMMIT

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12

    2、希望流量优先转发到本node的pod，如果不存在就转发到其他node的pod

    部署：

    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: nginx
  topologyKeys:
    - "kubernetes.io/hostname"
    - "*"

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34

    查看部署结果：

    [root@10-231-2-19 ~]# k get pod -o wide
NAME                                  READY   STATUS        RESTARTS   AGE   IP            NODE              NOMINATED NODE   READINESS GATES
nginx-deployment-j5wfx                1/1     Running       0          63m   10.244.2.51   10.231.2.21                  
nginx-deployment-pdhhz                1/1     Running       0          63m   10.244.1.45   10.231.2.20                  
nginx-deployment-x2fgf                1/1     Running       0          63m   10.244.5.70   10.231.2.46                  

[root@10-231-2-19 ~]# k get service
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes                    ClusterIP   10.96.0.1               443/TCP        99d
nginx                         ClusterIP   10.102.57.42            80/TCP         63m

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10

    查看带有nginx pod的node上的nginx service iptables规则：只设置了本node的pod转发

    我们查看了 10.231.2.20上的规则，其设置dnat规则刚好是本node上的10.244.1.45这个pod

    -A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx" -j KUBE-SEP-YTKAPZ6VEZPKTH4H
-A KUBE-SEP-YTKAPZ6VEZPKTH4H -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.244.1.45:80

    • 1
    • 2

    查看没有nginx pod的node上的nginx service iptables规则：设置了三个后端pod的均衡转发

    -A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-YTKAPZ6VEZPKTH4H
-A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-DCKS3M5ZM6QVK62Q
-A KUBE-SVC-2CMXP7HKUVJN7L6M -m comment --comment "default/nginx" -j KUBE-SEP-PP5KUAFXNANKDDU6
-A KUBE-SEP-PP5KUAFXNANKDDU6 -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.244.5.70:80
-A KUBE-SEP-YTKAPZ6VEZPKTH4H -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.244.1.45:80
-A KUBE-SEP-DCKS3M5ZM6QVK62Q -p tcp -m comment --comment "default/nginx" -m tcp -j DNAT --to-destination 10.244.2.51:80

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6

    原理总结

    • 使用serviceTopo之后，如果设置了只能就近转发，那么node上kube-proxy在生成iptables规则时，如果本node上有此后端pod不会生成多个负载均衡，而只生成定向转发到本node的pod的dnat规则。如果本node上没有此后端pod，那么就不会为service生成dnat规则
    • 使用serviceTopo之后，如果设置就近转发且如果node上没有对应后端的pod，则转发其它node上的pod，那么node上kube-proxy在生成iptables规则时，如果本node上有此后端pod不会生成多个负载均衡，而只生成定向转发到本node的pod的dnat规则。如果本node上没有此后端pod，那么会为service生成多个后端pod的dnat规则，也就是此时跟普通service转发一样

    注意：serviceTopo是依赖endpointslice来记录信息的

    我们部署了一个带serviceTopo的service，则会生成其对应的endpointslice，查看：
    kubectl get endpointslice nginx-ncf2b -o yaml:

    addressType: IPv4
apiVersion: discovery.k8s.io/v1beta1
endpoints:
- addresses:
  - 10.244.1.45
  conditions:
    ready: true
  targetRef:
    kind: Pod
    name: nginx-deployment-pdhhz
    namespace: default
    resourceVersion: "15352690"
    uid: c3e758f1-4ca7-442a-9761-fc780a117d4b
  topology:
    kubernetes.io/hostname: 10.231.2.20
- addresses:
  - 10.244.2.51
  conditions:
    ready: true
  targetRef:
    kind: Pod
    name: nginx-deployment-j5wfx
    namespace: default
    resourceVersion: "15352701"
    uid: 67b97b7a-7829-420b-b5e3-2182c2ebd785
  topology:
    kubernetes.io/hostname: 10.231.2.21
- addresses:
  - 10.244.5.70
  conditions:
    ready: true
  targetRef:
    kind: Pod
    name: nginx-deployment-x2fgf
    namespace: default
    resourceVersion: "15352705"
    uid: 7332ace0-c080-4570-8f8e-c8e114aad287
  topology:
    kubernetes.io/hostname: 10.231.2.46
kind: EndpointSlice
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2022-10-29T04:29:43Z"
  creationTimestamp: "2022-10-29T04:29:38Z"
  generateName: nginx-
  generation: 4
  labels:
    app: nginx
    endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
    kubernetes.io/service-name: nginx
  name: nginx-ncf2b
  namespace: default
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Service
    name: nginx
ports:
- name: ""
  port: 80
  protocol: TCP

    • 1
    • 2
    • 3
    • 4
    • 5
    • 6
    • 7
    • 8
    • 9
    • 10
    • 11
    • 12
    • 13
    • 14
    • 15
    • 16
    • 17
    • 18
    • 19
    • 20
    • 21
    • 22
    • 23
    • 24
    • 25
    • 26
    • 27
    • 28
    • 29
    • 30
    • 31
    • 32
    • 33
    • 34
    • 35
    • 36
    • 37
    • 38
    • 39
    • 40
    • 41
    • 42
    • 43
    • 44
    • 45
    • 46
    • 47
    • 48
    • 49
    • 50
    • 51
    • 52
    • 53
    • 54
    • 55
    • 56
    • 57
    • 58
    • 59
    • 60
    • 61
    • 62
  • 相关阅读:
    Leetcode46. Permutations | 模拟 | 递归 | 记忆化搜索
    使用 Spring Envers 的数据库事务审计
    什么是 DNS 泛洪攻击（DNS 泛洪）
    极客日报：罗永浩吐槽苹果新品：更丑更贵更胡来；苹果售卖145元擦屏布；MySQL 8.0.27 GA版本发布
    SpringBoot整合RabbitMQ，实现单机抢票系统
    先验 & 后验 & 似然估计
    Django--ORM 多表查询
    王道数据结构【链表】部分代码实现（C语言）
    ubuntu给终端加代理服务器
    LeetCode 630. 课程表 III - 优先队列&贪心
  • 原文地址：https://blog.csdn.net/qq_43684922/article/details/127586088