🐳 Section 14 · Question #10

What is Node in Kubernetes

A Node (Worker Node) is a physical bare-metal server or virtual machine in a Kubernetes cluster that provides the underlying compute, memory, storage, and networking capacity re...

🐳 Docker Kubernetes 📄 Source Markdown

🟢 Junior Level

30-Second Interview Answer

A Node (Worker Node) is a physical bare-metal server or virtual machine in a Kubernetes cluster that provides the underlying compute, memory, storage, and networking capacity required to run containerized Pods.

While the Control Plane acts as the brain and administrative center of the cluster, worker nodes are the “hands and feet” doing the actual work. Kubernetes aggregates the hardware resources of dozens or hundreds of nodes into a single unified elastic resource pool.

Real-World Analogy: The Cargo Harbor

Imagine an international shipping container terminal:

  • Control Plane (Dispatch Tower): Schedules shipments, tracks ship locations, and decides where containers should be placed.
  • Node (Cargo Ship): A physical vessel with a defined carrying capacity (CPU and RAM).
  • Pod (Container Group): Cargo loaded onto the deck of the ship.
  • Ship Captain (kubelet): Stationed on the ship, the captain inspects cargo lashings, monitors engine health, and reports back to the dispatch tower regarding available deck space.

3 Mandatory Daemons on Every Worker Node

  1. kubelet: The primary node agent. It communicates with kube-apiserver, accepts PodSpec assignments, directs the container runtime to launch or kill containers, executes health checks, and reports node heartbeat telemetry.
  2. kube-proxy: The network proxy daemon. It programs host kernel network packet filters (iptables or IPVS) to provide Layer 4 load balancing and route Service virtual IPs directly to Pod IPs.
  3. Container Runtime (CRI): The low-level runtime engine. Following the deprecation and removal of dockershim in Kubernetes 1.24, modern nodes run standard OCI/CRI-compliant runtimes—primarily containerd or CRI-O.

🟡 Middle Level

Resource Budgeting: Capacity vs Allocatable

A common operational misconception is assuming that 100% of a server’s physical RAM and CPU is available to Pods. Kubernetes protects the host OS from kernel lockups by strictly partitioning resources:

\[\mathbf{Allocatable} = \mathbf{Capacity} - \mathbf{kube\text{-}reserved} - \mathbf{system\text{-}reserved} - \mathbf{eviction\text{-}threshold}\] 
+--------------------------------------------------------------------------+
|                  Total Physical Node Capacity (e.g., 64 GiB RAM)         |
|                                                                          |
|  [ kube-reserved ]   -> Reserved for K8s daemons (kubelet, containerd)   |
|  [ system-reserved ] -> Reserved for Linux OS & system daemons (sshd)    |
|  [ hard-eviction ]   -> Buffer margin triggering eviction (e.g. 100 MiB) |
| ──────────────────────────────────────────────────────────────────────── |
|  [ Allocatable ]     -> ONLY THIS PORTION is available for Pod scheduling!|
+--------------------------------------------------------------------------+

[!IMPORTANT] The kube-scheduler makes scheduling decisions strictly against the Allocatable budget. If the sum of CPU/RAM requests of existing Pods exhausts Allocatable, the scheduler refuses to schedule additional Pods onto that node—even if physical RAM appears largely unconsumed in htop.

Node Conditions (Health Monitoring)

The kubelet reports node health flags to the Control Plane: | Condition | Healthy Value | Operational Meaning | |—|:—:|—| | Ready | True | The node is healthy, daemons are running, and it is ready to accept Pods. | | MemoryPressure | False | Node memory has dropped below eviction thresholds; scheduling is throttled. | | DiskPressure | False | Root disk or container image storage partition is running out of space. | | PIDPressure | False | Available Linux process IDs (pids_max) are nearing exhaustion. | | NetworkUnavailable| False | The cluster CNI network plugin has not configured the node’s routing table. |

Node Maintenance Lifecycle: Cordon, Drain, Uncordon

When performing hardware upgrades or OS kernel patches on a worker node:

  1. kubectl cordon <node> (Quarantine):
    • Marks the node as SchedulingDisabled.
    • Prevents kube-scheduler from assigning new Pods to this node; existing Pods continue running untouched.
  2. kubectl drain <node> --ignore-daemonsets --delete-emptydir-data (Eviction):
    • Safely and gracefully evicts all workloads, triggering graceful shutdown hooks and recreating the Pods on healthy neighboring nodes.
  3. kubectl uncordon <node> (Recommission):
    • Re-enables scheduling after the node reboots and health is verified.

🔴 Senior Level

Workload Placement: Taints, Tolerations & Node Affinity

Senior Engineers configure precise scheduling topologies (e.g., isolating GPU workloads or reserving high-memory nodes for databases):

     [ Standard Pod ]                   [ GPU Workload with Toleration ]
            │                                           │
            ▼                                           ▼
     ❌ Rejected!                              🟢 Successfully Scheduled!
            │                                           │
            └──────────────► [ Tainted Node ] ◄─────────┘
                             gpu=true:NoSchedule

1. Node Taints

Taints repel Pods that do not possess a matching toleration: kubectl taint nodes gpu-node-01 dedicated=gpu:NoSchedule

  • NoSchedule: Hard constraint for new Pods. Untolerated Pods are never scheduled here. Existing Pods remain running.
  • PreferNoSchedule: Soft constraint. The scheduler avoids this node unless no other compute exists in the cluster.
  • NoExecute: Eviction constraint. Any currently running Pod lacking a matching toleration is immediately evicted from the node.

2. Pod Tolerations

Allows a Pod to tolerate and be scheduled onto a tainted node:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

3. Node Affinity (Pod Attraction)

  • requiredDuringSchedulingIgnoredDuringExecution: Hard rule. If no node matches the label (e.g., topology.kubernetes.io/zone=us-east-1a), the Pod remains Pending.
  • preferredDuringSchedulingIgnoredDuringExecution: Soft rule. Assigns priority weights (1–100) to steer Pods toward preferred hardware without blocking deployment.

High-Scale Node Heartbeats: The Lease Object Architecture

Prior to Kubernetes 1.14, every kubelet transmitted full NodeStatus updates to kube-apiserver every 10 seconds.

  • In clusters with 2,000+ nodes, this generated immense write contention in etcd (a single Node object with container image lists and IPs can exceed 20 KB).
  • Modern Architecture:
    • The heavy Node object is updated only when node conditions change (e.g., once every minute).
    • Frequent 10-second heartbeats are transmitted via a lightweight Lease object in the kube-node-lease namespace.
    • If a node fails to renew its Lease for 40 seconds, the Controller Manager transitions the node to Ready=Unknown.

Kubelet Local Eviction Manager

When physical memory drops below hard eviction thresholds (e.g., memory.available < 100Mi):

  1. The kubelet acts locally and immediately without waiting for the Control Plane.
  2. It evicts Pods in strict reverse priority order based on Quality of Service (QoS) classes and memory consumption relative to requests:
    • BestEffort: Evicted first.
    • Burstable: Evicted second if consuming more memory than their declared requests.memory.
    • Guaranteed: Evicted last, only if no other workloads remain to be reclaimed.

4 Tricky Questions with In-Depth Answers

1. Why does kubectl drain <node> fail by default if there are Pods with emptyDir volumes or Bare Pods without controllers?

Answer: The drain command enforces safety checks to prevent unintentional, irreversible data loss:

  1. emptyDir Volumes: Data stored inside an emptyDir resides exclusively on that specific node’s local disk. Terminating the Pod destroys that local directory permanently. Hence, drain requires explicit acknowledgement via the --delete-emptydir-data flag.
  2. Bare Pods (Unmanaged Pods): If a Pod was deployed directly via kind: Pod rather than through a Deployment or StatefulSet, no controller exists to recreate it on another node. Evicting it deletes the workload permanently. To drain bare pods, --force is required.
  3. DaemonSets: DaemonSet Pods are pinned to every node. Evicting them to another node is impossible because that node already runs its own DaemonSet replica. Draining requires --ignore-daemonsets.

2. What happens to running Pods on a node if the kubelet daemon process crashes or freezes?

Answer:

  1. Container Processes Continue Running: The containerd runtime runs as an independent system daemon. Existing container processes and kernel routing (kube-proxy) continue executing and serving traffic without interruption.
  2. Heartbeat Timeout: The node stops renewing its Lease object. After 40 seconds, the Control Plane marks the node as Ready=Unknown.
  3. Eviction Grace Period: The pod-eviction-timeout timer starts (default: 5 minutes).
  4. Rescheduling: If kubelet fails to recover within 5 minutes, the Deployment controller assumes the node is lost and provisions replacement Pods on healthy nodes elsewhere in the cluster.
  5. StatefulSet Caveat: For StatefulSet Pods, replacement Pods are NOT automatically created to avoid split-brain data corruption on shared volumes until the dead node is confirmed deleted.

3. What is the fundamental difference between the Taint effects NoSchedule and NoExecute?

Answer:

  • NoSchedule: Acts exclusively as a filter during the kube-scheduler placement phase for future Pods. Pods that were already executing on the node prior to the taint being applied remain running undisturbed.
  • NoExecute: Operates retroactively on both future and existing Pods. Applying a NoExecute taint immediately triggers the eviction of all running Pods on that node that lack a matching toleration (or evicts them after their configured tolerationSeconds grace period expires).

4. How does modern autoscaling with Karpenter architecturally outperform traditional Kubernetes Cluster Autoscaler?

Answer:

  • Cluster Autoscaler (Legacy): Relies on cloud-provider Auto Scaling Groups (ASGs). When Pods are Pending, it scales a pre-defined ASG node pool of identical instance types (e.g., adding another m5.large). Provisioning takes 3 to 6 minutes, and workloads with varying CPU/GPU shapes suffer from resource fragmentation and wasted compute.
  • Karpenter: Bypasses cloud ASGs entirely and communicates directly with cloud provisioning APIs (e.g., AWS EC2 Fleet API). It inspects the aggregate resource requests, affinities, architectures (ARM vs x86), and topologies of all unscheduled Pods and provisions a custom, right-sized instance (or Spot fleet) in 30 to 45 seconds, reducing cloud infrastructure costs by up to 50%.

🎯 Interview Cheat Sheet

Summary of Worker Nodes

  • Definition: Physical or virtual worker host executing containerized Pods.
  • Core Node Stack: kubelet (agent), kube-proxy (L4 network routing), containerd / CRI-O (CRI runtime).
  • Allocatable Math: $\text{Allocatable} = \text{Capacity} - \text{kube/system reserves}$. Scheduling is based strictly on Allocatable!
  • Maintenance Steps: cordon (stop new pods) $\to$ drain (evict existing pods) $\to$ uncordon (reopen scheduling).
  • Placement Controls: Taints & Tolerations (repelling mechanism) vs Node Affinity (attracting mechanism).
  • Heartbeat Architecture: Lightweight Lease objects in kube-node-lease update every 10s to prevent etcd overload.

Taint Effects Comparison

| Taint Effect | Future Pods | Currently Running Pods | |—|—|—| | NoSchedule | ❌ Blocked from scheduling | 🟢 Untouched (Continue running) | | PreferNoSchedule | 🟡 Avoided if alternative nodes exist | 🟢 Untouched | | NoExecute | ❌ Blocked from scheduling | 🔴 Immediately evicted from the node! |

Red Flags (DO NOT Say)

  • ❌ “A Node is just another Docker container.” (A Node is the underlying OS host running the container runtime engine).
  • ❌ “All 64 GB of a 64 GB server is available for scheduling Pods.” (Fails to understand Allocatable and OS reserve margins).
  • ❌ “If kubelet crashes, all running containers die instantly.” (Containers run inside independent containerd processes and continue serving traffic).
  • ❌ “Taints pull Pods to a specific node.” (Taints repel Pods; Node Affinity attracts Pods).