EPIC - k8s Gateway

The k8s Application & API Gateway

EPIC is an Application and API Gateway designed for microservice applications hosted on Kubernetes. The EPIC platform creates Gateways that consist of microservice envoy proxy engines. Behind the scenes, the EPIC gateway is a purpose-built k8s cluster orchestrating network facing and cluster facing configuration necessary to automate the creation of Gateways at the request of the Gateway Controller.

Gateway Templates

EPIC creates Gateways and proxy instances in user-namespaces. Contained in each namespace is the primary configuration element of the gateways: the gateway template. By default there will be a choice of standard templates and additional templates can be added.

The template combines other configuration components such as IP address prefix ranges, DNS configuration, and certificates, with an Envoy configuration to create the required number of proxies required by the application. Once a Gateway is created, a specific configuration for that gateway is created. Editing this configuration changes the gateway dynamically.

Cluster Endpoints

Gateway creation is initiated by the Gateway Controller, this is a two step process. The first step creates the EPIC gateway and its instances when the gateway resource is created in the cluster. When route resources are added in the cluster, the node-controller communicates with the Endpoint controller in EPIC, dynamically crated endpoints for the proxy. This dynamic behavior allows endpoints to be managed using the k8s API from the cluster, for example, scaling a deployment will result in more endpoints being created in the cluster and EPIC, thereby scaling the application. These endpoints are displayed as routes in the Gateway Service Manager. Note that the references in the Gateway Service Manager match the references in the cluster making it easy to show how the paths to cluster resources have been created.

Gateway Configuration

A gateway template combines the Envoy configuration with the configuration of other components.

The template below is one of the defaults found in a new customer namespace. This is the complete template and includes system information.

epic.acnodal.io/owning-serviceprefix: v6this selects the pool of service prefixes that will be used to add external IP addresses to the gateway. This example is an IPv6 address. EPIC supports multiple address pools, in this case the address pool is global, user-namespace based pools can also be configured
spec: can-be-shared: yesThis flag indicates that multiple clusters can use this Gateway. When enabled in the Controller, the cluster creating the Gateway receives a sharing key in the gateway status. By adding this key to the GatewayClassConfig, creating a gateway in the cluster will not create a new gateway, instead it will add the endpoints to the existing shared gateway. This is how multi-cloud is provisioned.
spec: endpoint-templateThis template creates a DNS name that can be added to the DNS. The gateway configuration is updated to include the address and the generated name of the potential DNS record.
envoy-replica-count: 2Each Gateway consists of a number of instances of the Envoy proxy. In the future the the envoy-replica-count will also be configurable using the Gateway Controller Policy in the cluster. The EPIC cluster places each instance on a different cluster node, advertising both instances to our edge router using BGP, loadbalancing traffic between the instances and providing redundancy
envoy-templateA standard yaml based Envoy configuration
envoyResources: endpoints:This part of the configuration is dynamically updated by the endpoint controller. However additional static endpoints and clusters can be added to combine CDN resources with k8s services
envoyResources: clustersA combination of configuration elements, are shown, the cluster information is provided by the endpoint discovery controller, however other elements such as the lb_policy are configured here
envoyResources: listenersAgain a combination of configuration, routes are provided by the endpoint controller and the remaining configuration is provided in the template
secrets:Where TLS is used, the envoy configuration references the Secrets Discovery Server. The referenced secrets are contained in this configuration element. Secrets can be either global (wildcard for default https services) or located in the user-names. EPIC provides a secrets manager based upon Jetstack.
apiVersion: epic.acnodal.io/v1
kind: LBServiceGroup
  namespace: epic-epictest
  name: gatewayhttpsv6
    epic.acnodal.io/owning-account: epictest
    epic.acnodal.io/owning-serviceprefix: v6
    app.kubernetes.io/component: lbservicegroup
    app.kubernetes.io/part-of: epic
  can-be-shared: yes
    dnsName: "{{.LBName}}.{{.LBSGName}}.client.acnodal.io"
    recordTTL: 180
    recordType: A
  envoy-replica-count: 1
    envoyAPI: v3
    serialization: yaml
    nodeID: SET_BY_EPIC
      - name: SET_BY_EPIC
        value: |
          cluster_name: {{.ClusterName}}
          {{- if .Endpoints}}
          - lb_endpoints:
          {{- range .Endpoints}}
            - endpoint:
                    address: {{.Spec.Address}}
                    protocol: {{.Spec.Port.Protocol | ToUpper}}
                    port_value: {{.Spec.Port.Port}}
          {{- end}}
          {{- end}}          
      - name: SET_BY_EPIC
        value: |
          name: {{.ClusterName}}
          connect_timeout: 2s
          type: EDS
              resource_api_version: V3
                api_type: GRPC
                transport_api_version: V3
                - envoy_grpc:
                    cluster_name: eds-server
          lb_policy: ROUND_ROBIN
          - interval: 5s
            timeout: 5s
            no_traffic_interval: 5s
            unhealthy_threshold: 3
            healthy_threshold: 3
              path: /ready          
      - name: SET_BY_EPIC
        value: |
          name: {{.PortName}}
              port_value: {{.Port}}
              protocol: {{.Protocol | ToUpper}}
          - filters:
            - name: envoy.http_connection_manager
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: {{ .ServiceName }}
                  name: local_route
                  {{- range .Routes }}
                  {{- $route := .Spec.HTTP }}
                  {{- range $route.Hostnames | HostnameOrDefault }}
                  - name: "{{ . }}"
                    - "{{ . }}"
                    {{- if $route.Rules}}
                    {{- range $route.Rules}}
                    - route:
                          {{- range .BackendRefs}}
                          - name: {{ .Name }}
                            weight: {{ .Weight }}
                          {{- end }}
                          total_weight: {{ .BackendRefs | RefWeightsTotal }}
                      {{- if .Matches }}
                      {{- range .Matches }}
                        {{- if .Path }}
                        prefix: "{{ .Path.Value }}"
                        {{- end }}
                        {{- if .Headers }}
                        {{- range .Headers }}
                        - name: "{{ .Name }}"
                          {{- if .Type | HeaderTypeRegex }}
                              google_re2: {}
                              regex: "{{ .Value }}"
                          {{- else }}
                            exact: "{{ .Value }}"
                          {{- end }}
                        {{- end }}
                        {{- end }}
                      {{- end }}
                      {{- end }}
                    {{- end }}
                    {{- end }}
                  {{- end }}
                  {{- end }}
                - name: envoy.filters.http.router
              name: envoy.transport_socket.tls
                  "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
                      - name: gwtmpl-https-cert
                          resource_api_version: V3
                          ads: {}          
      - name: gwtmpl-https-cert
          name: gwtmpl-https-cert
          namespace: epic-epictest