云原生之旅 - 9）云原生时代网关的后起之秀Envoy Proxy 和基于Envoy 的 Emissary Ingress

前言

前一篇文章讲述了基于Nginx代理的Kuberenetes Ingress Nginx【云原生时代的网关 Ingress Nginx】这次给大家介绍下基于Envoy的 Emissary Ingress。

 

首先什么是Enovy？

Envoy 是由 Lyft 开源的高性能网络代理软件，后来捐赠给了 CNCF 基金会，已经毕业于CNCF。 相比于 Nginx、HAProxy 等经典代理软件，Envoy 具备丰富的可观察性和灵活的可扩展性，并且引入了基于 xDS API 的动态配置方案，Envoy 还提供了大量的开箱即用的 Filter 以满足各种场景下流量治理的需求。

Envoy 与 Nginx 代理的区别

• Envoy 对 HTTP/2 的支持比 Nginx 更好，支持包括 upstream 和 downstream在内的双向通信，而 Nginx 只支持 downstream 的连接。
• 高级负载均衡功能是免费的，Nginx 的高级负载均衡功能则需要商业版 Nginx Plus 支持。
• Envoy 支持热更新，Nginx 配置更新之后需要 Reload。
• Envoy 更贴近 Service Mesh 的使用习惯，Nginx 更贴近传统服务的使用习惯。

Envoy 有典型的两种工作模式。一种作为中心代理，代理集群的南北向流量，这种模式下，Envoy 一般就是负载均衡设备或者是 API 网关的基础数据面，比如 Ambassador 现在叫 Emissary，Gloo 都是新兴的开源的基于 Envoy 的开源网关。另一种模式，就是作为业务进程的 Sidecar，当有业务请求访问业务的时候，流量会被劫持到 Sidecar Envoy 当中，之后再被转发给业务进程，典型代表 Istio 和 Linkerd.

今天我们介绍的就是代理南北向流量的网关 Emissary Ingress（原名 Ambassador）。Emissary-ingress已经是CNCF的孵化项目，并且在去年被顶级服务网格项目Linkerd和Istio正式支持。如需集成参考文档。

 

关键词：基于Enovy的Emissary Ingress实践，Emissary Ingress入门，云原生网关Emissary Ingress，Emissary Ingress实践

 

为什么选择 Emissary Ingress

https://www.getambassador.io/docs/emissary/latest/about/alternatives/

https://www.getambassador.io/docs/emissary/latest/about/faq/#why-emissary-ingress

 

功能列表

常见的云原生网关功能都有，像流量管理，限流，熔断，canary release， authentication，详见下面列表。

Reference: https://github.com/emissary-ingress/emissary

• Manage ingress traffic with load balancing, support for multiple protocols (gRPC and HTTP/2, TCP, and web sockets), and Kubernetes integration
• Manage changes to routing with an easy to use declarative policy engine and self-service configuration, via Kubernetes CRDs or annotations
• Secure microservices with authentication, rate limiting, and TLS
• Ensure high availability with sticky sessions, rate limiting, and circuit breaking
• Leverage observability with integrations with Grafana, Prometheus, and Datadog, and comprehensive metrics support
• Enable progressive delivery with canary releases
• Connect service meshes including Consul, Linkerd, and Istio
• Knative serverless integration

 

安装

使用Terraform Helm Provider

从 emissary-ingress 2.1开始, 它把 CRDs 从Helm Charts移除了, 现在首先需要手动 apply CRDs。

?

kubectl apply -f https://app.getambassador.io/yaml/emissary/3.2.0/emissary-crds.yaml

所以我做了一个Helm Charts 专门装下CRDs，否则无法全流程安装自动化。

如果不了解Helm Chart 请参考这篇文章【Kubernetes时代的包管理工具 Helm】入门。

resource "helm_release" "emissary_crds" {
     name             = "emissary-crds"
     create_namespace = true # create emissary default namespace `emissary-system`
     namespace        = local.emissary_ns
     chart            = "../common/helm/repos/emissary-crds-8.2.0.tgz"
}

CRDs是默认装在`emissary-system` namespace下面的，不建议修改namespace，如果要在不同的Namespace下装多个Emissary ingress， 是可以共用这个CRDs的。

下面这部分是官方chart

# Install Emissary-ingress from Chart Repository
resource "helm_release" "emissary_ingress" {
  name             = "emissary-ingress"
  repository       = "https://app.getambassador.io"
  chart            = "emissary-ingress"
  version          = local.chart_version
  create_namespace = true
  namespace        = local.emissary_ns

  values = [
    templatefile("${local.common_yaml_d}/emissary-ingress-template.yaml", local.emissary_ingress_map)
  ]

  depends_on = [
    helm_release.emissary_crds
  ]
}

最后一部分，也是自制 chart 专门负责config

# This is for install Host/Listener/Mapping/TLSContext from a local custom chart
# also can upload chart to a bucket or a public github for install from a url
# e.g. [Publish to a GCS bucket](https://github.com/hayorov/helm-gcs)
resource "helm_release" "emissary_config" {
  name      = "emissary-config"
  namespace = local.emissary_ns
  chart     = "../common/helm/repos/emissary-config-8.2.0.tgz"

  values = [
    templatefile("${local.common_yaml_d}/emissary-listeners-template.yaml", local.emissary_listeners_map),
    local.emissary_config_yaml
  ]

  depends_on = [
    helm_release.emissary_ingress
  ]
}

locals 变量

locals {

  project_id     = "global-sre-dev"
  cluster_name   = "sre-gke"
  cluster_region = "us-central1"
  emissary_ns    = "emissary"
  chart_version  = "8.2.0"
  common_yaml_d  = "../common/helm/yamls"
  ambassador_id  = "ambassador"

  emissary_ingress_map = {
    ambassadorID          = local.ambassador_id
    loadBalancerIP        = "35.232.98.249" # Prepare a Static IP first instead to use Ephemeral
    replicaCount          = 2
    minReplicas           = 2
    maxReplicas           = 3
    canaryEnabled         = false # set to true in Prod
    logLevel              = "error" # valid log levels are error, warn/warning, info, debug, and trace
    endpointEnable        = true
    endpointName          = "my-resolver"
    diagnosticsEnable     = false
    clusterRequestTimeout = 120000 # milliseconds
  }

  emissary_listeners_map = {
    ambassadorID          = local.ambassador_id
    listenersEnabled      = true # custom listeners
  }
}

config文件

locals {
  emissary_config_yaml = <<-EOT
    hosts:
    - name: my-host-dev
      spec:
        ambassador_id: 
        - ${local.ambassador_id}
        hostname: '*.wadexu.cloud'
        requestPolicy:
          insecure:
            action: Redirect
        tlsContext:
          name: my-tls-context
        tlsSecret:
          name: tls-secret
          namespace: secret
    mappings:
    - name: my-nginx-mapping
      spec:
        ambassador_id:
        - ${local.ambassador_id}
        hostname: dev.wadexu.cloud
        prefix: /
        service: my-nginx.nginx:80

    tlscontexts:
    - name: my-tls-context
      spec:
        ambassador_id: 
        - ${local.ambassador_id}
        hosts:
        - "*.wadexu.cloud"
        min_tls_version: v1.2
  EOT
}

完整代码请参考 my repo

另外因为用的https，所以需要一个tls-secret 安装在secret ns下面

kubectl create secret -n secret tls tls-secret \
  --key ./xxx.key \
  --cert ./xxx.pem

Install from local， (Optional) 如果要学习自动化Terraform安装，请参考【部署Terrform基础设施代码的自动化利器 Atlantis】

cd terraform_helm_install/dev

terraform init
terraform plan 
terraform apply

Install result

% helm list -n emissary-system
NAME         	NAMESPACE      	REVISION	UPDATED                            	STATUS  	CHART              	APP VERSION
emissary-crds	emissary-system	1       	2022-10-20 10:09:30.72553 +0800 CST	deployed	emissary-crds-8.2.0	3.2.0         

% helm list -n emissary                             
NAME            	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                 	APP VERSION
emissary-config 	emissary 	1       	2022-10-20 10:31:24.819555 +0800 CST	deployed	emissary-config-8.2.0 	3.2.0      
emissary-ingress	emissary 	1       	2022-10-20 10:29:33.705888 +0800 CST	deployed	emissary-ingress-8.2.0	3.2.0   

 

使用 Kustomize

参考我的 quick start

如果不了解 Kustomize， 请移步我这篇文章【不能错过的一款 Kubernetes 应用编排管理神器 Kustomize】

 

一个集群安装多个Emissary Ingress

我这个例子 This example 展示了 multiple Emissary deployed in one cluster.

在一个集群里安装多个 Emissary 一定要设置 ambassador_id 并且替换 ClusterRoleBinding name， 否则资源冲突。

• emissary-ingress-init: CRDs will be installed.
• emissary-ingress-public: An emissary-ingress with allow list = all (face to internet).
• emissary-ingress-private: Another emissary-ingress with an allow list (restrict connection) installed in same cluster.

 

Test in local

# apply CRDs first
kustomize build emissary-ingress-init/sre-mgmt-dev > ~/init.yaml
kubectl apply -f ~/init.yaml

# deploy first public Emissary, this allow list = all, face to internet
kustomize build emissary-ingress-public/sre-mgmt-dev > ~/emissary_deploy1.yaml
kubectl apply -f ~/emissary_deploy1.yaml

# deploy second private Emissary with a restrict allow list to access
kustomize build emissary-ingress-private/sre-mgmt-dev > ~/emissary_deploy2.yaml
kubectl apply -f ~/emissary_deploy2.yaml

通过Terraform安装 Kustomize资源，请参考 my repo

如：

module "example_custom_manifests" {
  source  = "kbst.xyz/catalog/custom-manifests/kustomization"
  version = "0.3.0"

  configuration_base_key = "default"
  configuration = {
    default = {

      resources = [
        "${path.root}/../../infra/emissary-ingress-init/sre-mgmt-dev"
      ]

      common_labels = {
        "env" = "dev"
      }
    }
  }
}

 

Test

建一个nginx service 测试下

helm install my-nginx bitnami/nginx --set service.type="ClusterIP" -n nginx --create-namespace

curl

% curl https://dev.wadexu.cloud







Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.

For online documentation and support please refer to
nginx.org.

Commercial support is available at
nginx.com.

Thank you for using nginx.

 

FAQ

1. 这个error 代表 tls-secret 有问题，确保正确创建

error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

 

2. Connection refused, 最大的可能是 Listeners 没有配置好。

curl: (7) Failed to connect to dev.wadexu.cloud port 443 after 255 ms: Connection refused

 

3. CRDs 没创建。

│ Error: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "my-resolver" namespace: "emissary-system" from "": no matches for kind "KubernetesEndpointResolver" in version "getambassador.io/v2"
│ ensure CRDs are installed first, resource mapping not found for name: "ambassador" namespace: "emissary-system" from "": no matches for kind "Module" in version "getambassador.io/v2"
│ ensure CRDs are installed first]

注意: If helm provider > 2.7.0, plan will prompt this error. Workaround is apply CRDs first. `terraform apply -target helm_release.emissary_crds` 然后 apply 剩下的资源。

所以用helm provider <= 2.6.0一次性创建比较好。这个问题已经有人在github 提过issue了。

 

另外，TLSContext 里面的 secret_namespacing 不work，issue， 但不影响，我的例子把tls-secret放在kind: Host下面。

 

 

感谢阅读，如果您觉得本文的内容对您的学习有所帮助，您可以打赏和推荐，您的鼓励是我创作的动力。