• kubernetes中ingress控制器traefik获取真实客户源IP

    一.现象

    公司kubenetes生产环境使用的ingress控制器是traefik，默认是通过deployment部署的，现在研发上反馈不能获取客户的真实源IP地址，通过x_forward_for获取的IP地址都是kubernetes集群内部的IP地址。

    二.解决思路

    通过查找traefik的官方文档

    Traefik EntryPoints Documentation - Traefik

    traefik传输客户源地址到后端需要配置Forwarded Headers参数，用于x_forwarded_for保存客户源IP地址

    具体配置如下

     yaml格式，用于启用的时候指定配置文件

    1. ## Static configuration
    2. entryPoints:
    3.   web:
    4.     address: ":80"
    5.     forwardedHeaders:
    6.       insecure: true

    TOML格式，用于traefik启动的时候指定配置文件，这个文件在kubernetes中可以以comfigmap存在

    1. ## Static configuration
    2. [entryPoints]
    3.   [entryPoints.web]
    4.     address = ":80"
    5.  
    6.     [entryPoints.web.forwardedHeaders]
    7.       insecure = true

    命名行格式

    1. --entryPoints.web.address=:80
    2. --entryPoints.web.forwardedHeaders.insecure

    三.问题处理

    1.公司现有环境配置

    1. ---
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   annotations:
    6.     meta.helm.sh/release-name: traefik
    7.     meta.helm.sh/release-namespace: default
    8.   labels:
    9.     app.kubernetes.io/instance: traefik
    10.     app.kubernetes.io/managed-by: Helm
    11.     app.kubernetes.io/name: traefik
    12.     helm.sh/chart: traefik-9.11.0
    13.   name: traefik
    14.   namespace: default
    15.   resourceVersion: '505763774'
    16. spec:
    17.   progressDeadlineSeconds: 600
    18.   replicas: 6
    19.   revisionHistoryLimit: 10
    20.   selector:
    21.     matchLabels:
    22.       app.kubernetes.io/instance: traefik
    23.       app.kubernetes.io/name: traefik
    24.   strategy:
    25.     rollingUpdate:
    26.       maxSurge: 1
    27.       maxUnavailable: 1
    28.     type: RollingUpdate
    29.   template:
    30.     metadata:
    31.       creationTimestamp: null
    32.       labels:
    33.         app.kubernetes.io/instance: traefik
    34.         app.kubernetes.io/managed-by: Helm
    35.         app.kubernetes.io/name: traefik
    36.         helm.sh/chart: traefik-9.11.0
    37.     spec:
    38.       containers:
    39.         - args:
    40.             - '--global.checknewversion'
    41.             - '--global.sendanonymoususage'
    42.             - '--entryPoints.traefik.address=:9000/tcp'
    43.             - '--entryPoints.web.address=:8000/tcp'
    44.             - '--entryPoints.websecure.address=:8443/tcp'
    45.             - '--api.dashboard=true'
    46.             - '--ping=true'
    47.             - '--providers.kubernetescrd'
    48.             - '--providers.kubernetesingress'
    49.           image: 'traefik:2.3.3'
    50.           imagePullPolicy: IfNotPresent
    51.           livenessProbe:
    52.             failureThreshold: 3
    53.             httpGet:
    54.               path: /ping
    55.               port: 9000
    56.               scheme: HTTP
    57.             initialDelaySeconds: 10
    58.             periodSeconds: 10
    59.             successThreshold: 1
    60.             timeoutSeconds: 2
    61.           name: traefik
    62.           ports:
    63.             - containerPort: 9000
    64.               name: traefik
    65.               protocol: TCP
    66.             - containerPort: 8000
    67.               name: web
    68.               protocol: TCP
    69.             - containerPort: 8443
    70.               name: websecure
    71.               protocol: TCP
    72.           readinessProbe:
    73.             failureThreshold: 1
    74.             httpGet:
    75.               path: /ping
    76.               port: 9000
    77.               scheme: HTTP
    78.             initialDelaySeconds: 10
    79.             periodSeconds: 10
    80.             successThreshold: 1
    81.             timeoutSeconds: 2
    82.           resources: {}
    83.           securityContext:
    84.             capabilities:
    85.               drop:
    86.                 - ALL
    87.             readOnlyRootFilesystem: true
    88.             runAsGroup: 65532
    89.             runAsNonRoot: true
    90.             runAsUser: 65532
    91.           terminationMessagePath: /dev/termination-log
    92.           terminationMessagePolicy: File
    93.           volumeMounts:
    94.             - mountPath: /data
    95.               name: data
    96.             - mountPath: /tmp
    97.               name: tmp
    98.       dnsPolicy: ClusterFirst
    99.       restartPolicy: Always
    100.       schedulerName: default-scheduler
    101.       securityContext:
    102.         fsGroup: 65532
    103.       serviceAccount: traefik
    104.       serviceAccountName: traefik
    105.       terminationGracePeriodSeconds: 60
    106.       volumes:
    107.         - emptyDir: {}
    108.           name: data
    109.         - emptyDir: {}
    110.           name: tmp
    111. status:
    112.   availableReplicas: 6
    113.   conditions:
    114.     - lastTransitionTime: '2022-10-10T07:58:50Z'
    115.       lastUpdateTime: '2022-10-10T07:58:50Z'
    116.       message: Deployment has minimum availability.
    117.       reason: MinimumReplicasAvailable
    118.       status: 'True'
    119.       type: Available
    120.     - lastTransitionTime: '2020-11-25T22:53:59Z'
    121.       lastUpdateTime: '2022-11-17T10:44:40Z'
    122.       message: ReplicaSet "traefik-54bf67c74d" has successfully progressed.
    123.       reason: NewReplicaSetAvailable
    124.       status: 'True'
    125.       type: Progressing
    126.   observedGeneration: 13
    127.   readyReplicas: 6
    128.   replicas: 6
    129.   updatedReplicas: 6
    130.  
    131. ---
    132. apiVersion: v1
    133. kind: Service
    134. metadata:
    135.   annotations:
    136.     meta.helm.sh/release-name: traefik
    137.     meta.helm.sh/release-namespace: default
    138.   labels:
    139.     app.kubernetes.io/instance: traefik
    140.     app.kubernetes.io/managed-by: Helm
    141.     app.kubernetes.io/name: traefik
    142.     helm.sh/chart: traefik-9.11.0
    143.   name: traefik
    144.   namespace: default
    145.   resourceVersion: '505762379'
    146. spec:
    147.   clusterIP: 10.96.252.108
    148.   externalTrafficPolicy: Cluster
    149.   ports:
    150.     - name: web
    151.       nodePort: 30079
    152.       port: 80
    153.       protocol: TCP
    154.       targetPort: web
    155.   selector:
    156.     app.kubernetes.io/instance: traefik
    157.     app.kubernetes.io/name: traefik
    158.   sessionAffinity: None
    159.   type: NodePort
    160. status:
    161.   loadBalancer: {}
    162.

    发现trafik启动的参数里面没有指定配置文件，也没有以comfigmap形式存在，所以只能考虑通过添加启动参数的方式（即命令行方式）解决问题

    2.解决问题

    在deployment部署中的traefik启动参数中添加 - '--entrypoints.web.forwardedHeaders.insecure'和- '--entrypoints.websecure.forwardedHeaders.insecure'启动参数，其中标红的web和websecure根据--entryPoints.web.address=:8000/tcp和--entryPoints.websecure.address=:8443/tcp要保持一致，有可能不一样。

    修改后的derloyment的yaml文件如下：

    1. ---
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   annotations:
    6.     meta.helm.sh/release-name: traefik
    7.     meta.helm.sh/release-namespace: default
    8.   labels:
    9.     app.kubernetes.io/instance: traefik
    10.     app.kubernetes.io/managed-by: Helm
    11.     app.kubernetes.io/name: traefik
    12.     helm.sh/chart: traefik-9.11.0
    13.   name: traefik
    14.   namespace: default
    15.   resourceVersion: '505763774'
    16. spec:
    17.   progressDeadlineSeconds: 600
    18.   replicas: 6
    19.   revisionHistoryLimit: 10
    20.   selector:
    21.     matchLabels:
    22.       app.kubernetes.io/instance: traefik
    23.       app.kubernetes.io/name: traefik
    24.   strategy:
    25.     rollingUpdate:
    26.       maxSurge: 1
    27.       maxUnavailable: 1
    28.     type: RollingUpdate
    29.   template:
    30.     metadata:
    31.       creationTimestamp: null
    32.       labels:
    33.         app.kubernetes.io/instance: traefik
    34.         app.kubernetes.io/managed-by: Helm
    35.         app.kubernetes.io/name: traefik
    36.         helm.sh/chart: traefik-9.11.0
    37.     spec:
    38.       containers:
    39.         - args:
    40.             - '--global.checknewversion'
    41.             - '--global.sendanonymoususage'
    42.             - '--entryPoints.traefik.address=:9000/tcp'
    43.             - '--entryPoints.web.address=:8000/tcp'
    44.             - '--entryPoints.websecure.address=:8443/tcp'
    45.             - '--api.dashboard=true'
    46.             - '--ping=true'
    47.             - '--providers.kubernetescrd'
    48.             - '--providers.kubernetesingress'
    49.             - '--entrypoints.web.forwardedHeaders.insecure'
    50.             - '--entrypoints.websecure.forwardedHeaders.insecure'
    51.           image: 'traefik:2.3.3'
    52.           imagePullPolicy: IfNotPresent
    53.           livenessProbe:
    54.             failureThreshold: 3
    55.             httpGet:
    56.               path: /ping
    57.               port: 9000
    58.               scheme: HTTP
    59.             initialDelaySeconds: 10
    60.             periodSeconds: 10
    61.             successThreshold: 1
    62.             timeoutSeconds: 2
    63.           name: traefik
    64.           ports:
    65.             - containerPort: 9000
    66.               name: traefik
    67.               protocol: TCP
    68.             - containerPort: 8000
    69.               name: web
    70.               protocol: TCP
    71.             - containerPort: 8443
    72.               name: websecure
    73.               protocol: TCP
    74.           readinessProbe:
    75.             failureThreshold: 1
    76.             httpGet:
    77.               path: /ping
    78.               port: 9000
    79.               scheme: HTTP
    80.             initialDelaySeconds: 10
    81.             periodSeconds: 10
    82.             successThreshold: 1
    83.             timeoutSeconds: 2
    84.           resources: {}
    85.           securityContext:
    86.             capabilities:
    87.               drop:
    88.                 - ALL
    89.             readOnlyRootFilesystem: true
    90.             runAsGroup: 65532
    91.             runAsNonRoot: true
    92.             runAsUser: 65532
    93.           terminationMessagePath: /dev/termination-log
    94.           terminationMessagePolicy: File
    95.           volumeMounts:
    96.             - mountPath: /data
    97.               name: data
    98.             - mountPath: /tmp
    99.               name: tmp
    100.       dnsPolicy: ClusterFirst
    101.       restartPolicy: Always
    102.       schedulerName: default-scheduler
    103.       securityContext:
    104.         fsGroup: 65532
    105.       serviceAccount: traefik
    106.       serviceAccountName: traefik
    107.       terminationGracePeriodSeconds: 60
    108.       volumes:
    109.         - emptyDir: {}
    110.           name: data
    111.         - emptyDir: {}
    112.           name: tmp
    113. status:
    114.   availableReplicas: 6
    115.   conditions:
    116.     - lastTransitionTime: '2022-10-10T07:58:50Z'
    117.       lastUpdateTime: '2022-10-10T07:58:50Z'
    118.       message: Deployment has minimum availability.
    119.       reason: MinimumReplicasAvailable
    120.       status: 'True'
    121.       type: Available
    122.     - lastTransitionTime: '2020-11-25T22:53:59Z'
    123.       lastUpdateTime: '2022-11-17T10:44:40Z'
    124.       message: ReplicaSet "traefik-54bf67c74d" has successfully progressed.
    125.       reason: NewReplicaSetAvailable
    126.       status: 'True'
    127.       type: Progressing
    128.   observedGeneration: 13
    129.   readyReplicas: 6
    130.   replicas: 6
    131.   updatedReplicas: 6
    132.  
    133. ---
    134. apiVersion: v1
    135. kind: Service
    136. metadata:
    137.   annotations:
    138.     meta.helm.sh/release-name: traefik
    139.     meta.helm.sh/release-namespace: default
    140.   labels:
    141.     app.kubernetes.io/instance: traefik
    142.     app.kubernetes.io/managed-by: Helm
    143.     app.kubernetes.io/name: traefik
    144.     helm.sh/chart: traefik-9.11.0
    145.   name: traefik
    146.   namespace: default
    147.   resourceVersion: '505762379'
    148. spec:
    149.   clusterIP: 10.96.252.108
    150.   externalTrafficPolicy: Cluster
    151.   ports:
    152.     - name: web
    153.       nodePort: 30079
    154.       port: 80
    155.       protocol: TCP
    156.       targetPort: web
    157.   selector:
    158.     app.kubernetes.io/instance: traefik
    159.     app.kubernetes.io/name: traefik
    160.   sessionAffinity: None
    161.   type: NodePort
    162. status:
    163.   loadBalancer: {}

    3.验证测试

    通过和研发和业务验证测试，通过x_forwarded_for获取的客户端源Ip地址格式如下

     其中第一个公网IP地址是客户的真实IP地址，其他几个是经过的k8s节点和pod的IP地址，至此解决业务上获取真实客户IP地址的需求。

  • 相关阅读:
    从github上下载下来的代码下载依赖提示： An unknown git error occurred
    【计算机网络-自顶向下方法】应用层（SMTP、POP3、DNS）
    2022百度之星初赛第三场--字符计数
    【问题解决】日志服务Logtail采集容器标准输出过滤某个容器
    ARM机下编辑安装mysql错误处理
    SkyWalking安装部署
    选择排序超详细讲解C语言
    Python语言的12个基础知识点小结
    什么是顶级域名？如何获得顶级域名解析方案
    在CentOS 7.7 x86_64上安装python3.11.0实录
  • 原文地址：https://blog.csdn.net/rendongxingzhe/article/details/127932143