🐳 Section 14 · Question #8

What is Kubernetes and Why is It Needed

While Docker packages an application into an isolated container on a single host machine, Kubernetes orchestrates hundreds or thousands of containers running across dozens of ph...

🐳 Docker Kubernetes 📄 Source Markdown

🟢 Junior Level

30-Second Interview Answer

Kubernetes (K8s) is the industry-standard, open-source container orchestration platform designed to automate the deployment, horizontal scaling, self-healing, load balancing, and lifecycle management of containerized applications across a distributed cluster of nodes.

While Docker packages an application into an isolated container on a single host machine, Kubernetes orchestrates hundreds or thousands of containers running across dozens of physical or virtual servers:

  • Declarative Desired State: You declare what the system should look like in YAML manifests (e.g., “The order service must always maintain 5 healthy replicas”).
  • Automated Self-Healing: If a container crashes, the node runs out of memory, or a physical server dies, Kubernetes automatically restarts the failed container or reschedules it onto a healthy node within seconds.

Real-World Analogy: The Symphony Conductor

  • A Docker container is like an individual musician—a virtuoso violinist playing their assigned solo.
  • Kubernetes is the symphony orchestra conductor (the Greek word kubernētēs translates directly to “helmsman” or “pilot”):
    • The conductor assigns each musician to their designated seat on the stage (Scheduling).
    • If a violinist suddenly falls ill mid-performance, the conductor seamlessly signals an understudy to take their place (Self-Healing).
    • If the concert hall fills to capacity and more acoustic volume is needed, the conductor signals extra brass players to join in (Autoscaling).

5 Core Capabilities of Kubernetes

  1. Service Discovery and Load Balancing: Assigns persistent virtual IPs (ClusterIP) and DNS names to dynamic pods, evenly distributing incoming network traffic.
  2. Horizontal Autoscaling: Automatically increases or decreases pod counts based on CPU/RAM metrics or custom business events (via Horizontal Pod Autoscaler).
  3. Zero-Downtime Rolling Updates: Upgrades container versions incrementally, routing traffic away from terminating pods to newly verified pods with automated rollback on failure.
  4. Configuration and Secret Governance: Injects database passwords, TLS certificates, and environment properties without modifying or rebuilding container images.
  5. Storage Orchestration: Automatically attaches local, cloud-native (AWS EBS, GCP Persistent Disk, Azure Managed Disks), or distributed storage volumes (Ceph, NFS).

🟡 Middle Level

Cluster Architecture: Control Plane vs Worker Nodes

A Kubernetes cluster is divided into two distinct functional layers:

+-------------------------------------------------------------------------------+
|                       CONTROL PLANE (Cluster Brain)                           |
|                                                                               |
|  [ kube-apiserver ] <---> [ etcd (Raft DB) ]                                  |
|        ^       ^                                                              |
|        |       +-------- [ kube-scheduler ]                                   |
|        v                                                                      |
|  [ kube-controller-manager ] <---> [ cloud-controller-manager ]               |
+-------------------------------------------------------------------------------+
       | (HTTPS / gRPC)
       v
+-------------------------------------------------------------------------------+
|                       WORKER NODES (Compute Layer)                            |
|                                                                               |
|  [ Node 1 ]                          [ Node 2 ]                               |
|  ├── kubelet                         ├── kubelet                              |
|  ├── kube-proxy                      ├── kube-proxy                           |
|  ├── Container Runtime (containerd)  ├── Container Runtime (containerd)       |
|  └── Pod A | Pod B                   └── Pod C | Pod D                        |
+-------------------------------------------------------------------------------+

1. Control Plane Components (The Management Plane)

  • kube-apiserver: The central REST API gateway of the cluster. It validates and configures data for API objects (Pods, Services, Deployments) and is the only component that interacts directly with the etcd datastore.
  • etcd: A highly consistent, distributed key-value store implementing the Raft consensus algorithm. It holds the authoritative source of truth for the entire cluster state.
  • kube-scheduler: Watches for unscheduled pods and assigns them to optimal worker nodes based on resource requests, hardware constraints, node affinity, taints, and tolerations.
  • kube-controller-manager: Runs continuous reconciliation loops (Reconciliation Loop) that drive the current cluster state toward the desired state (e.g., Node Lifecycle Controller, ReplicaSet Controller, EndpointSlice Controller).
  • cloud-controller-manager: Integrates with cloud provider APIs to provision managed load balancers, cloud storage volumes, and node routing tables.

2. Worker Node Components (The Execution Plane)

  • kubelet: The primary node agent that registers the node with the API server. It receives PodSpec manifests and commands the container runtime to launch, monitor, and clean up containers.
  • kube-proxy: Manages host network rules (iptables or IPVS) to provide Layer 4 TCP/UDP load balancing and service abstraction across dynamic pod IPs.
  • Container Runtime: The low-level software implementing the Container Runtime Interface (CRI)—typically containerd or CRI-O—responsible for pulling images and executing containers via runc.

Core Kubernetes API Objects

| Object | Primary Responsibility | |—|—| | Pod | The atomic deployable compute unit (one or more containers sharing network and storage namespaces). | | Deployment | Declarative controller for stateless applications; manages ReplicaSets, rolling updates, and rollbacks. | | Service | Stable Layer 4 virtual IP (ClusterIP) and internal DNS entry that load-balances traffic across healthy Pods. | | Ingress | Layer 7 HTTP/HTTPS reverse proxy providing path-based and host-based routing, SSL/TLS termination. | | ConfigMap / Secret | Decouples external configuration and sensitive credentials (API keys, passwords) from container images. | | PersistentVolumeClaim (PVC) | Declarative request for persistent block or file storage that survives Pod restarts and node evictions. | | Namespace | Provides virtual cluster partitioning for multi-tenancy, RBAC isolation, and resource quotas. |

🔴 Senior Level

The Reconciliation Loop & Event-Driven Architecture

Kubernetes is architected on control theory and feedback loops: \(\text{Desired State (Stored in etcd)} \longleftrightarrow \text{Actual State (Reported by Nodes)}\)

  1. Watch API & Informers: Controllers do not poll the API server using heavy periodic requests. Instead, they open long-lived HTTP streaming connections (HTTP chunked watch) managed by cached Informers.
  2. Reflector & Delta FIFO Queue: When a resource state changes in etcd, the API server streams a delta event to the controller’s local queue.
  3. Reconcile() Method: The controller compares current reality with desired state (e.g., “etcd requires 5 replicas, but kubelet reports only 4 running”). The controller then issues declarative mutations to the API server to bridge the gap.

Admission Control Pipeline: The Lifecycle of an API Request

When an operator or CI/CD runner executes kubectl apply -f deployment.yaml:

+-----------------------------------------------------------------------------------+
|                       API Server Admission Control Pipeline                       |
|                                                                                   |
|   1. Authentication (TLS Certs, Bearer Tokens, OIDC/OAuth2)                       |
|          │                                                                        |
|          ▼                                                                        |
|   2. Authorization (RBAC: Verifies Role / ClusterRole permissions)                |
|          │                                                                        |
|          ▼                                                                        |
|   3. Mutating Admission Webhooks (Dynamic mutation: Istio sidecar / Vault secrets)|
|          │                                                                        |
|          ▼                                                                        |
|   4. Schema Validation (Verifies OpenAPI/JSON schema syntax)                      |
|          │                                                                        |
|          ▼                                                                        |
|   5. Validating Admission Webhooks (Enforces policies: OPA Gatekeeper / Kyverno)  |
|          │                                                                        |
|          ▼                                                                        |
|   6. ETCD Persistence (Atomic commit with incremented resourceVersion)            |
+-----------------------------------------------------------------------------------+

etcd Quorum Mechanics & Storage Performance (Raft Consensus)

A Senior Engineer must understand the mathematical invariants governing etcd:

  • Quorum Formula: Requires a strict majority to commit state: \(\text{Quorum} = \lfloor N/2 \rfloor + 1\)
  • Cluster Sizing: Must always use odd node counts ($2F + 1$, where $F$ is fault tolerance):
    • 3-node cluster: Quorum is 2. Tolerates 1 node failure.
    • 5-node cluster: Quorum is 3. Tolerates 2 node failures.
  • Disk I/O Sensitivity: etcd relies on sequential WAL (Write-Ahead Logging) commits. It strictly requires fsync latencies of $< 10\text{ ms}$ (high-performance NVMe SSDs). If disk latency degrades, Raft leader heartbeats drop, triggering frequent leader elections that paralyze the API server.

4 Tricky Questions with In-Depth Answers

1. What happens to running applications if all Control Plane master nodes (kube-apiserver and etcd) simultaneously crash?

Answer: Active user traffic and running containers continue functioning without interruption! The Control Plane is responsible for cluster state changes, scheduling, and API requests, but it is not in the data path of application traffic:

  • Worker nodes execute containers via containerd independently of the master nodes.
  • Layer 4 packet routing rules (iptables / IPVS) previously programmed by kube-proxy remain loaded in the host Linux kernel.
  • Ingress controllers (like Nginx or Traefik) and internal Services continue routing traffic to healthy Pod IPs.
  • What stops working: You cannot deploy new applications, run kubectl commands, autoscale pods (HPA freezes), or reschedule pods if a worker node crashes.

2. How does kube-scheduler place pods onto nodes, and what are its two core phases?

Answer: The scheduler executes a two-phase filtering and prioritization cycle for each unscheduled Pod:

  1. Filtering Phase (Predicates): Evaluates hard constraints to eliminate ineligible nodes:
    • Does the node have sufficient unallocated CPU and RAM to satisfy the Pod’s requests (PodFitsResources)?
    • Does the node match nodeSelector and NodeAffinity requirements?
    • Does the node have any Taints that the Pod lacks matching Tolerations for?
  2. Scoring Phase (Priorities): Ranks remaining candidate nodes from 0 to 100 using weighted scoring plugins:
    • ImageLocality: Favors nodes that have already downloaded the container image.
    • NodeResourcesBalancedAllocation: Favors nodes that balance CPU and memory usage evenly.
    • PodTopologySpread: Spreads replicas across different availability zones to ensure disaster resilience. The node with the highest aggregate score is chosen for Pod binding.

3. Why is it critically important to specify both requests and limits in Pod specs, and what happens when they are omitted?

Answer:

  • requests: Used exclusively by the kube-scheduler during node placement. It guarantees that the node has that amount of capacity reserved. If omitted, requests default to 0; the scheduler could pack 50 resource-intensive Java microservices onto a single small node, starving the system.
  • limits: Enforced at runtime by the Linux kernel (Cgroups):
    • Exceeding CPU limit causes the kernel to throttle CPU cycles (CFS Bandwidth Throttling), introducing severe application latency without killing the process.
    • Exceeding Memory limit triggers an immediate kernel kill via the Linux OOM-killer (Exit Code 137 / OOMKilled). If limits are omitted, a memory-leaking Java application can consume all physical RAM on the server, causing the host Linux kernel to kill vital host daemons like kubelet or containerd.

4. What is Split-Brain in an etcd cluster, and why is a 4-node cluster less resilient than a 3-node cluster?

Answer: A Split-Brain occurs when a network partition divides a cluster into isolated groups, each attempting to accept writes independently, causing data divergence and corruption. Raft prevents this by requiring a strict majority quorum ($\lfloor N/2 \rfloor + 1$):

  • In a 3-node cluster, quorum is $2$. If 1 node fails or gets partitioned, the remaining 2 nodes have quorum and continue processing writes (tolerates 1 failure).
  • In a 4-node cluster, quorum is $\lfloor 4/2 \rfloor + 1 = 3$. If a network partition splits the cluster into two equal halves of $2$ nodes and $2$ nodes, neither partition can achieve a quorum of 3. The entire cluster freezes all write operations! Thus, a 4-node cluster still tolerates only 1 node failure, but adds higher network consensus overhead and a greater vulnerability to split-brain deadlocks than a 3-node cluster.

🎯 Interview Cheat Sheet

Summary of Kubernetes

  • Definition: Enterprise-grade open-source container orchestration platform managing declarative desired state.
  • Control Plane: kube-apiserver (API gateway), etcd (Raft state store), kube-scheduler (node binding), kube-controller-manager (reconciliation loops).
  • Worker Node: kubelet (agent), kube-proxy (L4 routing), containerd (container runtime).
  • Core Loop: Continuously reconciles Actual State reported by Kubelets with Desired State in etcd.
  • Core Primitives: Pod (smallest unit) $\to$ Deployment (stateless updates) $\to$ Service (L4 routing) $\to$ Ingress (L7 edge routing).

Managed Kubernetes (EKS / GKE / AKS) vs Self-Hosted (Kubeadm)

| Feature | Managed Kubernetes (EKS/GKE/AKS) | Self-Hosted (Kubeadm / On-Premise) | |—|—|—| | Control Plane Management| Handled entirely by Cloud Provider (SLA 99.95%+) | Handled by in-house DevOps/SRE team | | etcd Backups & Maintenance| Automated snapshotting, scaling, and recovery | Manual scripts, monitoring, and DR testing | | Cluster Upgrades | Single-click / automated rolling control plane upgrades | Complex manual control plane certificate & binary upgrades | | Operational Overhead | Pay small cluster fee or worker node cost | Requires 3–5 dedicated full-time platform engineers |

Red Flags (DO NOT Say)

  • ❌ “Kubernetes is a direct replacement for Docker.” (Kubernetes is an orchestrator that coordinates container runtimes like containerd/Docker).
  • ❌ “If the master node crashes, all user traffic immediately stops.” (Existing pods and kernel iptables rules on worker nodes continue running and serving traffic).
  • ❌ “You should run etcd on 2 nodes to save costs.” (Even numbers cannot form a reliable majority quorum during network partitions).
  • ❌ “Kubernetes should be used for every simple monolithic application.” (Introduces massive operational complexity with zero business value for simple setups).