How does scaling work in Kubernetes

Scaling in Kubernetes is the automated or manual dynamic adjustment of compute resources to handle shifting application traffic.

🟢 Junior Level

30-Second Interview Answer

Scaling in Kubernetes is the automated or manual dynamic adjustment of compute resources to handle shifting application traffic.

Kubernetes operates across three independent scaling tiers:

Horizontal Pod Autoscaling (HPA): Adding or removing Pod replicas (e.g., scaling from 2 to 10 instances). This is the primary and most robust scaling pattern for stateless microservices.
Vertical Pod Autoscaling (VPA): Dynamically increasing or decreasing the CPU and Memory limits allocated to individual container processes.
Cluster Autoscaling (Cluster Autoscaler / Karpenter): Automatically provisioning and adding new physical or virtual Worker Nodes when the cluster runs out of capacity to host new Pods.

Real-World Analogy: The Supermarket Checkout Area

Think of a busy supermarket front end:

Horizontal Scaling (HPA): Opening additional checkout registers staffed by extra cashiers as customer lines grow (more replicas).
Vertical Scaling (VPA): Giving a single cashier a faster barcode scanner, a cash recycler, and a faster POS terminal (more CPU and RAM per process).
Cluster Autoscaling: When all existing checkout counter spaces are occupied, knocking down a wall to expand the building floor space to fit new registers (adding more worker nodes).

Manual vs Automated Scaling

Manual Imperative Scaling:

kubectl scale deployment order-service --replicas=5

Automated Scaling (HPA):

# Scale between 2 and 10 replicas when average CPU crosses 60%
kubectl autoscale deployment order-service --cpu-percent=60 --min=2 --max=10

🟡 Middle Level

The 3-Tier Kubernetes Scaling Taxonomy

How HPA Calculates Replica Counts: Mathematical Model

The HPA controller queries the Kubernetes Metrics API every 15 seconds (default horizontal-pod-autoscaler-sync-period) and computes desired replicas using the formula:

\[\mathbf{DesiredReplicas} = \left\lceil \mathbf{CurrentReplicas} \times \left( \frac{\mathbf{CurrentMetricValue}}{\mathbf{TargetMetricValue}} \right) \right\rceil\]

Concrete Example:

Current Replicas: 2
Target CPU Utilization: 50%
Actual Current CPU Utilization: 80%
Calculation: $\lceil 2 \times (80 / 50) \rceil = \lceil 3.2 \rceil =$ 4 Replicas.

[!IMPORTANT] HPA strictly requires resources.requests! Utilization percentages are calculated strictly relative to the declared requests.cpu. If requests are omitted from the container specification, Metrics Server cannot compute the utilization percentage, causing the HPA controller to fail with ScalingActive=False.

Mitigating Oscillation (Flapping Effect)

During spiky traffic, an autoscaler can encounter “flapping”: traffic spikes for 5 seconds $\to$ HPA adds 10 Pods $\to$ traffic drops $\to$ HPA terminates 8 Pods $\to$ JVM has no time to warm up $\to$ remaining 2 Pods crash under load.

Kubernetes stabilizes this behavior using the behavior policy block:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0    # Scale up immediately on traffic spikes
      policies:
      - type: Percent
        value: 100                    # Double replicas within 15 seconds
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 minutes of steady low traffic before scaling down!
      policies:
      - type: Pods
        value: 1                      # Terminate at most 1 Pod per minute
        periodSeconds: 60

🔴 Senior Level

Why Scaling Java by Memory is a Fatal Anti-Pattern

A classic Senior-level system design trap:

“Can we configure HPA for a Spring Boot service targeting 75% memory utilization (type: Resource, name: memory)?”

HotSpot JVM Mechanics:

JVM Memory Reservation: Upon boot, the JVM claims virtual pages from the host operating system up to -Xmx (or MaxRAMPercentage).
Delayed OS Memory Reclamation: When garbage collectors (G1, ZGC) reclaim unreferenced objects in Eden and Old Gen, the underlying Linux kernel memory pages (Resident Set Size — RSS) are NOT immediately returned to the host OS.
The HPA Feedback Trap: The container runtime reports container_memory_working_set_bytes hovering steadily around 80–90%, even when business throughput drops to zero!
Runaway Scale-Out: The HPA interprets this as memory starvation and continually adds Pods until hitting maxReplicas. The cluster becomes clogged with idle, memory-bloated Pods, causing massive cloud bill overruns.
- Rule: Scale Java microservices strictly on CPU utilization or Custom/External metrics (RPS, Latency, Kafka Consumer Lag).

KEDA: Event-Driven Autoscaling & Scale-to-Zero

Standard HPA can only scrape internal cluster metrics and cannot scale workloads down to zero. KEDA (Kubernetes Event-driven Autoscaling) extends Kubernetes:

Evaluates external message brokers: Apache Kafka Consumer Lag, RabbitMQ Queue Length, AWS SQS, Azure Service Bus.
Scale-to-Zero ($0 \to 1$): When a queue is empty, KEDA scales the Deployment to 0 replicas. Upon message arrival, it immediately triggers the creation of 1 replica to begin processing.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-kafka-scaler
spec:
  scaleTargetRef:
    name: order-consumer-deployment
  minReplicaCount: 0                  # Scale to Zero support!
  maxReplicaCount: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka:9092
      consumerGroup: order-workers
      topic: new-orders
      lagThreshold: "50"              # Add 1 Pod for every 50 unread messages

Node Autoscaling: Karpenter vs Classic Cluster Autoscaler

In-Place Pod Resize (Kubernetes 1.27+ / 1.29 Beta)

Historically, changing CPU or RAM via VPA required terminating the Pod (RestartPolicy: Recreate). The InPlacePodVerticalScaling feature enables resizing container resources on the fly without restarting the JVM process by dynamically writing new bounds to Linux kernel cgroups.procs, eliminating connection drops.

4 Tricky Questions with In-Depth Answers

1. What happens if you attach both HPA (on CPU) and VPA (in automatic mode `UpdateMode: "Auto"`) to the same Deployment simultaneously?

Answer: A catastrophic Control Loop Conflict (Feedback Thrashing) will destroy cluster stability:

When traffic surges, CPU utilization spikes to 90%.
HPA detects high utilization and schedules additional Pod replicas ($2 \to 6$).
Simultaneously, VPA concludes that the current Pods are under-provisioned; it evicts and restarts them with larger CPU allocations ($1\text{ Core} \to 4\text{ Cores}$).
With 4 CPU cores per Pod, the relative CPU utilization drops to 25%.
In response, HPA triggers an aggressive scale-down ($6 \to 2$).
Both controllers fight indefinitely, causing non-stop Pod restarts, connection drops, and cascading downtime.
Rule: Never combine HPA and VPA on the same resource metric. You may pair HPA on Custom Metrics (RPS/Kafka lag) with VPA on Memory.

2. Why do Java Spring Boot applications autoscaled by HPA often experience 503 errors and latency spikes during the first 30 seconds of new replicas starting?

Answer: This is caused by JVM Warmup and JIT Compilation Latency:

When HPA spawns a new Pod, the JVM starts in interpreted bytecode mode. C1/C2 JIT compilers have not yet compiled hot paths into native machine instructions, classes are still loading, and HikariCP connection pools and application caches are uninitialized.
The instant the Pod’s readinessProbe succeeds, Kubernetes routes an equal share of cluster traffic to this “cold” Pod.
The cold Pod experiences 100% CPU spikes from simultaneous request processing and JIT compilation. Inbound TCP sockets saturate Tomcat queues, timing out and returning HTTP 503 / 504 errors. Production Mitigations:
- Migrate to Spring Boot 3 AOT / GraalVM Native Image (50 ms boot time, zero JIT warmup).
- Configure a Service Mesh (Envoy / Istio) with Slow Start Window (linearly ramping up traffic weight to new Pods over 60 seconds).
- Configure custom warmup endpoints before marking the Readiness probe healthy.

3. How does HPA handle Pods that are in an `Unready` or failing state during metric calculations?

Answer: The HPA controller includes protective filtering to prevent statistical skew:

If a Pod is in Unready status or its metrics cannot be collected (e.g., it is still booting), the HPA completely discards it from the calculation of average utilization.
Including cold Pods with 0% CPU would artificially suppress the average utilization metric, preventing HPA from scaling up when existing Pods are drowning.
If all Pods fail to report metrics, HPA freezes the replica count at its current value to prevent ungrounded scale-down events.

4. What is the difference between `Resource`, `Pods`, `Object`, and `External` metrics in the `autoscaling/v2` API?

Answer:

Resource: Standard container metrics collected directly from Linux cgroups via Metrics Server (strictly cpu or memory), evaluated as averageUtilization (% of requests) or averageValue.
Pods: Custom metrics describing Pod properties aggregated across all Pods in the target Deployment (e.g., average HTTP requests per second per Pod scraped via Prometheus Adapter).
Object: Metrics describing a separate, distinct Kubernetes resource within the same namespace (e.g., tracking the count of active connections on an Ingress Controller).
External: Metrics originating from cloud services or third-party platforms outside the Kubernetes cluster (e.g., AWS SQS queue depth, GCP Pub/Sub backlog, or Datadog monitor values).

🎯 Interview Cheat Sheet

Summary of Kubernetes Scaling

HPA: Horizontal pod scaling; standard for stateless microservices.
VPA: Vertical pod resource resizing (CPU/RAM).
Cluster Autoscaler / Karpenter: Provisions and removes worker nodes when Pods are Pending.
HPA Requirement: Metrics Server must be deployed, and Pods must declare resources.requests.
Java Scaling Rule: Never scale Java on Memory (JVM doesn’t release RSS memory quickly); scale on CPU, RPS, or Kafka lag.
Oscillation Protection: Configure behavior.scaleDown.stabilizationWindowSeconds (300s).
KEDA: Event-driven scaling on message queues with Scale-to-Zero capability.

Karpenter vs Classic Cluster Autoscaler

Red Flags (DO NOT Say)

❌ “I configure HPA for my Java microservices based on 80% RAM utilization.” (JVM memory retention causes HPA to permanently lock at maxReplicas).
❌ “HPA can calculate CPU percentages without declaring resources.requests.” (Impossible; requests serve as the 100% calculation baseline).
❌ “Running HPA and VPA simultaneously on CPU is best practice.” (Triggers a destructive feedback loop of non-stop Pod restarts).
❌ “Cluster Autoscaler provisions new nodes when cluster CPU reaches 90%.” (It triggers strictly when Pods fail to schedule and land in Pending state).

What is HorizontalPodAutoscaler — Dedicated HPA deep dive
What is ReplicaSet — How replica scaling is enacted
What is Node in Kubernetes — Worker node capacity and allocatable resources
What is Readiness Probe — Gating traffic to newly scaled Pods