🐳 Section 14 · Question #20

What is Ingress in Kubernetes

The core conceptual architecture relies on two distinct elements:

🐳 Docker Kubernetes 📄 Source Markdown

🟢 Junior Level

30-Second Summary

Ingress is a specialized Kubernetes API object that manages external inbound HTTP and HTTPS traffic (OSI Layer 7) into cluster services. It provides a single consolidated public entry point, domain- and path-based request routing, and centralized SSL/TLS termination.

The core conceptual architecture relies on two distinct elements:

  1. Ingress Resource (Manifest): A declarative YAML document defining routing rules (e.g., “Route requests for api.company.com/orders to the order-service“). On its own, this YAML is passive metadata and performs no packet routing.
  2. Ingress Controller (Execution Engine): An active reverse proxy daemon (such as NGINX, Traefik, HAProxy, or Envoy) running within the cluster. It watches the Kubernetes API Server for Ingress resources and dynamically updates its internal routing and proxy tables.

Real-World Analogy

Ingress is like the reception lobby of a corporate commercial office tower:

  • The skyscraper has a single street address and a central revolving door entrance.
  • A visitor enters the reception lobby and states their destination: “I am visiting Accounting” (request for /billing) or “I am here for Human Resources” (request for /hr).
  • The receptionist (Ingress Controller) checks building policies and directs the visitor to the appropriate elevator, floor, and office suite (internal ClusterIP Service).

Basic Ingress Manifest (networking.k8s.io/v1)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx # Binds to a specific Ingress Controller
  tls:
  - hosts:
    - api.company.com
    secretName: company-tls-cert # SSL certificate stored in a Kubernetes Secret
  rules:
  - host: api.company.com
    http:
      paths:
      - path: /orders
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 8080
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 8080

🟡 Middle Level

Economics and Architecture: Ingress vs. Service LoadBalancer

Without Ingress (Expensive & Difficult to Scale):
Client ──► Cloud Load Balancer 1 ($25/month) ──► Order Service (L4 TCP)
Client ──► Cloud Load Balancer 2 ($25/month) ──► User Service (L4 TCP)
Client ──► Cloud Load Balancer 3 ($25/month) ──► Payment Service (L4 TCP)

With Ingress (Cost-Effective, Centralized, L7 Routing):
Client ──► SINGLE Cloud Load Balancer ($25/month) ──► [ Ingress Controller (NGINX/Envoy) ]
                                                            │
                                  ┌─────────────────────────┼─────────────────────────┐
                                  ▼                         ▼                         ▼
                            Order Service             User Service             Payment Service
                           (ClusterIP:8080)          (ClusterIP:8080)         (ClusterIP:8080)
  • Layer 4 vs. Layer 7: A standard Service type: LoadBalancer operates at Layer 4 (TCP/UDP). It has no awareness of HTTP headers, cookies, URL paths, or TLS hostnames.
  • Ingress operates at Layer 7 (HTTP/HTTPS): It inspects application payloads, terminates TLS centrally, rewrites request paths, validates JWT headers, enforces rate limits, and routes traffic based on URL path prefixes and hostnames.

Path Matching Algorithms (pathType)

The networking.k8s.io/v1 specification mandates declaring an explicit pathType:

  • Exact: Matches the URL path strictly with case sensitivity. /orders matches only /orders; a request to /orders/123 returns 404 Not Found.
  • Prefix: Matches URL path segments split by /. /orders matches both /orders and /orders/123/items, but ignores /orders-archive.
  • ImplementationSpecific: Delegates matching behavior to the underlying controller implementation (e.g., regex pattern matching in NGINX Ingress).

Automated TLS Management via Cert-Manager

In production environments, TLS certificates are issued and renewed automatically using Cert-Manager with Let’s Encrypt:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod" # Triggers ACME challenge
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - shop.example.com
    secretName: shop-tls-secret # Cert-Manager populates and auto-renews this Secret

🔴 Senior Level

The “NGINX Reload Storm” Problem & The Envoy Alternative

The widely used NGINX Ingress Controller exhibits a known architectural scalability constraint in large, dynamic clusters:

  1. How NGINX Operates: Whenever Pod replicas change (via HPA scale out/in or rolling deployments), the controller generates a new monolithic nginx.conf file using Go text templates and issues a process reload signal: nginx -s reload.
  2. The Reload Storm: In clusters with hundreds of microservices undergoing continuous deployments, NGINX reloads every 2–3 seconds. During worker process restarts, NGINX resets internal caches, delays HTTP/2 multiplexing, and pauses WebSocket connections, generating severe latency spikes and transient 502 Bad Gateway / 504 Gateway Timeout errors.
  3. The Solution (Envoy xDS Streaming API): Modern ingress solutions built on Envoy (Contour, Envoy Gateway, Istio IngressGateway) utilize the streaming gRPC xDS protocol (EDS/CDS/LDS). Envoy receives endpoint and routing updates dynamically in memory in 2–5 milliseconds without process restarts, configuration reloads, or socket drops.

The Evolution: Kubernetes Gateway API (K8s 1.28+ GA)

Created in 2015, the classic Ingress API suffered from critical design limitations: vendor-specific annotation sprawl (nginx.ingress.kubernetes.io/*), lack of native TCP/UDP/gRPC support, and tangled role boundaries.

The Gateway API decouples responsibilities across organizational roles:

  • GatewayClass: Managed by Infrastructure / Cloud Providers (defines the proxy controller type).
  • Gateway: Managed by Cluster Administrators (defines the public IP, listening ports 80/443, and TLS certificates).
  • HTTPRoute / GRPCRoute: Managed by Application Developers (defines service path routing and traffic weighting):
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: order-route
spec:
  parentRefs:
  - name: production-gateway # Attaches to shared infrastructure gateway
  hostnames:
  - "api.company.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /orders
    backendRefs:
    - name: order-service
      port: 8080
      weight: 90
    - name: order-service-canary
      port: 8080
      weight: 10 # Native traffic splitting without vendor annotations!

4 Tricky Questions

1. What happens if you apply an Ingress manifest in a cluster that has no Ingress Controller deployed?

Answer: The manifest is successfully validated by kube-apiserver and persisted in etcd (the kubectl apply command succeeds). However, no traffic routing occurs:

  • The ADDRESS column in kubectl get ingress remains blank indefinitely.
  • No cloud load balancers or proxy routes are provisioned.
  • The Ingress resource remains a passive database record until an active Ingress Controller (e.g., ingress-nginx) is installed and configured to watch that ingressClassName.

2. How does the NGINX Ingress Controller handle Canary deployments, and what is the rule precedence when combining canary-weight and canary-by-header?

Answer: A canary deployment is declared as a second Ingress manifest with the annotation nginx.ingress.kubernetes.io/canary: "true". The controller generates a secondary upstream in the NGINX configuration. Rule Precedence:

  1. canary-by-header (Highest Priority): If a request contains the matching header (e.g., X-Canary: always), it is unconditionally routed to the canary upstream, completely bypassing percentage weights.
  2. canary-by-cookie (Medium Priority): If no matching header is present, the cookie value is evaluated next.
  3. canary-weight (Lowest Priority): Evaluated strictly as a fallback. Requests that match neither header nor cookie rules are distributed based on randomized percentage weighting.

3. Why do WebSocket connections fail through NGINX Ingress after 60 seconds of inactivity by default, and how is this fixed?

Answer: WebSockets begin with an HTTP Upgrade handshake and subsequently operate as a long-lived bidirectional TCP stream:

  • By default, NGINX Ingress sets both proxy-read-timeout and proxy-send-timeout to 60s.
  • If neither client nor server transmits data across the socket for 60 seconds, NGINX closes the TCP connection.
  • Resolution: Configure extended timeouts in the Ingress annotations: 
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

    Additionally, implement application-level heartbeat Ping/Pong frames (e.g., in Spring WebSocket or Netty) to keep the connection actively streaming.

4. What security vulnerability is associated with the configuration-snippet annotation in NGINX Ingress, and why is it disabled by default in modern cloud distributions?

Answer: The nginx.ingress.kubernetes.io/configuration-snippet annotation allows developers to inject raw NGINX directive strings directly into the generated nginx.conf.

  • Vulnerabilities (CVE-2021-25742 / CVE-2023-5043): A tenant with permissions to create an Ingress in an isolated namespace can inject arbitrary NGINX directives or Lua code. This allows reading private TLS keys belonging to other namespaces, extracting controller ServiceAccount tokens from /var/run/secrets/, or triggering Server-Side Request Forgery (SSRF) across the internal network.
  • As a security baseline, modern distributions set allow-snippet-annotations: "false" in the Ingress Controller’s ConfigMap to disallow custom snippet injection.

🎯 Interview Cheat Sheet

Core Ingress Architecture

  • Layer 7 Entry Point: Operates at OSI Layer 7 (HTTP/HTTPS); routes on hostnames and paths.
  • Resource vs. Controller: Manifest YAML is passive metadata; the Ingress Controller (NGINX/Envoy) actively proxies traffic.
  • Cost Reduction: Consolidates many internal ClusterIP microservices behind one shared cloud load balancer.
  • TLS Termination: Centralized SSL offloading using Kubernetes Secrets; automated via Cert-Manager.
  • NGINX Reload Limitation: High-frequency pod churn causes nginx -s reload storms; Envoy xDS solves this with lock-free dynamic memory updates.
  • Future Direction: Kubernetes Gateway API replaces Ingress with typed, role-oriented routing resources (HTTPRoute).

Comparison: Ingress vs. Service LoadBalancer

| Feature | Service (type: LoadBalancer) | Ingress | | :— | :— | :— | | OSI Layer | Layer 4 (TCP / UDP) | Layer 7 (HTTP / HTTPS / gRPC) | | Routing Granularity| IP and Port only | Hostname (host), Path (path), Headers | | TLS Termination | Managed per-service or external | Centralized at cluster boundary | | Cloud Cost | Incurs cost per Service | One cloud LB serves hundreds of services |

Red Flags (What NOT to Say)

  • ❌ “Ingress is a Layer 4 load balancer for routing raw TCP database connections.” — Ingress is strictly Layer 7 (HTTP/HTTPS/gRPC).
  • ❌ “Applying an Ingress YAML manifest automatically creates a functional routing proxy without a controller.” — An Ingress Controller daemon must be running to process rules.
  • ❌ “Every microservice in our cluster has its own public Cloud LoadBalancer Service.” — A major architectural anti-pattern leading to bloated cloud expenses.