How to organize rolling update in Kubernetes

A Rolling Update is the default deployment strategy for Kubernetes Deployment workloads. It incrementally replaces old Pod instances with new ones progressively in small batches...

🟢 Junior Level

30-Second Summary

A Rolling Update is the default deployment strategy for Kubernetes Deployment workloads. It incrementally replaces old Pod instances with new ones progressively in small batches, ensuring continuous service availability with Zero Downtime.

Instead of terminating all running containers simultaneously (the destructive Recreate strategy), Kubernetes spins up one or more new pods.
It waits until the new pods pass their readinessProbe.
Only after new replicas are verified ready does Kubernetes begin gracefully terminating older pods.
Throughout the entire process, client traffic is continuously served without disruption.

Real-World Analogy

A Rolling Update is like changing tires on a moving multi-axle freight truck or train:

The truck has 8 wheels.
Mechanics elevate one axle, unbolt the old wheel, mount the new tire, verify torque wrench tension (readinessProbe), and lower it back onto the highway.
Then they proceed to the next axle.
The truck never stops moving and continues delivering cargo. The alternative (Recreate) is cutting the engine, removing all 8 wheels at once, and leaving the truck propped on cinder blocks (total business outage).

Basic Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Allow at most 1 temporary extra pod (up to 4 + 1 = 5 pods)
      maxUnavailable: 0  # Zero pods allowed unavailable (at least 4 healthy pods at all times)
  selector:
    matchLabels:
      app: payment
  template:
    metadata:
      labels:
        app: payment
    spec:
      containers:
      - name: payment-app
        image: payment-service:2.1.0 # Specific semantic version tag (NEVER :latest!)
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          periodSeconds: 5

🟡 Middle Level

The Mathematics of `maxSurge` and `maxUnavailable`

Workload rollout behavior is governed by two parameters expressed as integers or percentages:

Baseline Replicas: spec.replicas = 4
maxSurge: 1       -> Peak allowed pods during rollout: 4 + 1 = 5 pods
maxUnavailable: 0 -> Minimum required responsive pods: 4 - 0 = 4 pods

Step-by-Step Rollout Progression:

Step 1: Kubernetes spawns 1 new Pod running version v2. Total cluster pods = 5 (4 v1 + 1 v2).
Step 2: The v2 Pod passes its readinessProbe. The Service adds it to Endpoints, routing live traffic to it.
Step 3: The controller terminates 1 old v1 Pod. Total cluster pods return to 4 (3 v1 + 1 v2).
Step 4: This batch cycle repeats iteratively until all 4 pods run v2, leaving 0 pods on v1.

Lifecycle Management via `kubectl rollout`

Kubernetes provides built-in commands to monitor, pause, and revert deployments:

# 1. Trigger an update by updating the container image:
kubectl set image deployment/payment-service payment-app=payment-service:2.2.0

# 2. Stream real-time rollout status:
kubectl rollout status deployment/payment-service

# 3. Pause an ongoing rollout (e.g., to validate telemetry on a canary pod):
kubectl rollout pause deployment/payment-service

# 4. Resume deployment rollout:
kubectl rollout resume deployment/payment-service

# 5. Inspect revision history:
kubectl rollout history deployment/payment-service

# 6. INSTANT ROLLBACK to the previous stable revision during an outage:
kubectl rollout undo deployment/payment-service

Under the Hood: The Role of ReplicaSets

A Deployment does not manage individual Pods directly; it orchestrates underlying ReplicaSet objects:

When the spec.template changes, the Deployment controller creates a new ReplicaSet (e.g., payment-78d4f8).
It progressively increments replicas in the new ReplicaSet while decrementing replicas in the old ReplicaSet (payment-59c6b8).
Once finished, the previous ReplicaSet remains preserved at replicas: 0. This enables kubectl rollout undo to revert traffic in less than a second simply by re-scaling the preserved ReplicaSet!

🔴 Senior Level

Version Coexistence & The “Expand / Contract” Pattern

During a rolling update, Version 1 (old) and Version 2 (new) execute concurrently against the database for several minutes:

Request A may hit v1, while Request B hits v2.
If a developer drops a table column or renames a JSON property in v2, remaining v1 pods will immediately throw runtime exceptions and crash.

Database Evolution: The Expand and Contract Pattern:

Phase 1 (Expand): Execute a Liquibase/Flyway migration: ADD the new column as NULLABLE. Version v1 continues running normally, ignoring the new column.
Phase 2 (Rollout): Execute the Rolling Update for v2. Version v2 dual-writes to both the old and new columns.
Phase 3 (Contract): After 100% rollout verification, execute a cleanup migration to drop the obsolete column.

Production Stalling Pitfalls: Quotas and PDBs

1. Deadlock via ResourceQuota Limits

If a Namespace has a hard quota of 10 pods, and replicas: 10 is active:

Deploying with maxSurge: 2 instructs Kubernetes to spawn an 11th and 12th pod.
The admission controller rejects pod creation with exceeded quota.
New pods sit permanently in Pending, freezing the rollout until progressDeadlineSeconds expires.
Solution: In quota-constrained namespaces, configure maxSurge: 0, maxUnavailable: 1 (evicts an old pod first to free quota before creating a new one).

2. Deadlock via PodDisruptionBudget (PDB)

If a PDB requires minAvailable: 100% on a 4-replica deployment while maxUnavailable: 25% is declared:

The Deployment controller cannot terminate an old pod because doing so violates the PDB constraint. The rollout deadlocks completely.

The 4-Pillar Zero-Downtime Architecture for Java/Spring Boot

To guarantee zero 502 Bad Gateway errors during rollouts:

maxUnavailable: 0: Never terminate an old pod until a replacement replica is confirmed operational.
Robust readinessProbe: Route traffic to Spring Boot only after bean wiring and cache priming are complete.
preStop: sleep 10: Delays SIGTERM by 10 seconds to allow cluster-wide kube-proxy iptables propagation to drain.
Spring Boot Graceful Shutdown: Enable server.shutdown: graceful and configure spring.lifecycle.timeout-per-shutdown-phase: 20s so Tomcat completes active transactions before container death.

4 Tricky Questions

1. Why does using the Docker image tag `:latest` (e.g., `image: my-app:latest`) completely break Kubernetes automated Rolling Updates?

Answer: The Deployment controller detects updates by hashing the spec.template (generating the pod-template-hash label):

If you push a new image build to your registry using the existing tag my-app:latest and run kubectl apply -f deployment.yaml, the text content of the manifest remains byte-for-byte identical.
Because the pod template hash has not changed, the Deployment controller determines that no configuration change occurred and does not trigger a Rolling Update.
Active pods continue running the stale code. To trigger automated rollouts reliably, always specify unique semantic version tags (:2.1.0) or immutable container digest hashes (@sha256:...).

2. What happens to `ConfigMap` and `Secret` resources when you run `kubectl rollout undo`?

Answer: They do NOT roll back! This is a major operational trap:

Kubernetes Deployment revision history tracks changes made exclusively to the Pod template specification (spec.template).
External resources like ConfigMap and Secret are independent objects without native revision linkage to the Deployment.
Running kubectl rollout undo rolls the container image back to the previous revision, but the container will mount the CURRENT, modified ConfigMap! If the old application binary is incompatible with the new configuration schema, it will crash immediately.
Mitigation: Use GitOps tooling (Helm, ArgoCD) or append content hashes to ConfigMap names (e.g., app-config-a8f3d1), updating the Deployment’s volume reference on each change (Immutable ConfigMaps).

3. What is the purpose of `spec.progressDeadlineSeconds`, and what happens if a deployment exceeds it during a failed rollout?

Answer:

progressDeadlineSeconds (default: 600 seconds / 10 minutes) specifies the maximum time Kubernetes waits for an update to demonstrate verifiable progress (pods launching, passing readiness checks).
If new pods become trapped in CrashLoopBackOff or ImagePullBackOff for longer than this duration:
- The Deployment controller updates its status condition Progressing to False with the reason ProgressDeadlineExceeded.
- Kubernetes DOES NOT perform an automated rollback! The rollout halts in its current intermediate state (retaining healthy old pods and broken new pods), signaling alerts to Prometheus or CI/CD pipelines to initiate manual or scripted intervention.

4. How does the `RollingUpdate` strategy differ between a `Deployment` and a `StatefulSet`?