🐳 Section 14 · Question #13

What is ReplicaSet

A ReplicaSet (RS) is a core Kubernetes system controller whose sole responsibility is to guarantee that a specified number of identical Pod replicas are running and available at...

🐳 Docker Kubernetes 📄 Source Markdown

🟢 Junior Level

30-Second Interview Answer

A ReplicaSet (RS) is a core Kubernetes system controller whose sole responsibility is to guarantee that a specified number of identical Pod replicas are running and available at any given moment.

If a developer declares replicas: 3:

  • The ReplicaSet controller continuously monitors the cluster: “How many healthy Pods matching my selector currently exist?”
  • If a Pod crashes, runs out of memory, or its host worker node burns out, the ReplicaSet immediately spawns a replacement Pod.
  • If extra matching Pods are spawned manually, the ReplicaSet terminates the surplus Pods to enforce the exact count of 3.

Real-World Analogy: The Supermarket Cashier Supervisor

Imagine the floor manager at a large supermarket:

  • Store policy mandates that there must always be exactly 4 checkout lanes open.
  • If a cashier falls sick or leaves for lunch, the supervisor immediately pulls a standby employee from the breakroom to open a register.
  • If customer foot-traffic drops and 5 cashiers are sitting at registers, the supervisor dismisses the extra cashier back to inventory restocking.

Basic ReplicaSet Manifest Example

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: order-service-rs
spec:
  replicas: 3              # Target desired replica count
  selector:
    matchLabels:
      app: order-service   # Governs Pods possessing this label
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-app
        image: order-service:1.0
        ports:
        - containerPort: 8080

[!IMPORTANT] Production Rule: In real-world enterprise deployments, you virtually never declare a raw ReplicaSet directly. Instead, you always declare a Deployment, which manages underlying ReplicaSets automatically to provide rolling updates, pause/resume, and instant rollbacks.

🟡 Middle Level

The Core Engine: The Reconciliation Loop

A ReplicaSet executes a continuous feedback control loop inside the kube-controller-manager:

+-------------------------------------------------------+
|        Read desired count from spec.replicas (e.g. 3) |
+---------------------------┬---------------------------+
                            │
                            ▼
+-------------------------------------------------------+
|       Count actual healthy Pods matching selector     |
+---------------------------┬---------------------------+
                            │
            ┌───────────────┴───────────────┐
            │                               │
      actual < desired                actual > desired
            │                               │
            ▼                               ▼
[ API Call: Spawn new Pod ]    [ API Call: Terminate surplus Pod ]

ReplicaSet vs Deployment: Why Deployments Rule

Why does industry standard mandate using Deployment rather than raw ReplicaSet?

Operational Feature ReplicaSet Deployment
Enforces $N$ Pod Replicas 🟢 Yes 🟢 Yes (via underlying RS)
Self-Healing on Node Failure 🟢 Yes 🟢 Yes (via underlying RS)
Zero-Downtime Rolling Updates ❌ No 🟢 Yes
Declarative Rollback (rollout undo) ❌ No 🟢 Yes
Release History & Revisions ❌ No 🟢 Yes

How Deployments Orchestrate ReplicaSets During Rolling Updates

When you update a container image in a Deployment from v1 to v2:

  1. The Deployment does NOT mutate the existing ReplicaSet. Instead, it generates a second, brand-new ReplicaSet for v2.
  2. The Deployment incrementally scales down RS-v1 ($3 \to 2 \to 1 \to 0$) while simultaneously scaling up RS-v2 ($0 \to 1 \to 2 \to 3$).
  3. The previous RS-v1 is retained in etcd with replicas: 0. If a defect is detected in production, executing kubectl rollout undo executes instantaneously by scaling RS-v1 back to 3 and RS-v2 down to 0!

Set-Based Selectors vs Legacy ReplicationControllers

The original Kubernetes v1.0 predecessor, ReplicationController, only supported simple equality-based label matching (app = order-service). Modern ReplicaSet supports powerful Set-Based Selectors (matchExpressions):

selector:
  matchExpressions:
  - key: environment
    operator: In
    values: [production, staging]
  - key: tier
    operator: NotIn
    values: [frontend]
  - key: release-stage
    operator: Exists

🔴 Senior Level

Pod Adoption and Orphaning Mechanics

A ReplicaSet binds to Pods dynamically using label evaluation rather than static references:

  1. Adoption:
    • If an unmanaged standalone Pod is created manually (kubectl run temp-pod --labels="app=order-service"), the ReplicaSet controller detects it, claims it toward its quota, and injects an ownerReferences block into the Pod’s metadata pointing to the ReplicaSet’s unique UID with controller: true.
  2. Orphaning:
    • Deleting a ReplicaSet with the cascading orphan flag (kubectl delete rs my-rs --cascade=orphan) deletes the controller record in etcd, but leaves all underlying Pods running indefinitely as independent standalone Pods.

Deployment Collision Prevention: pod-template-hash

To prevent different ReplicaSets belonging to the same Deployment from fighting over shared labels like app: order-service, the Deployment controller hashes the Pod template content (using the FNV-1a 32-bit hashing algorithm) and injects a unique label:

labels:
  app: order-service
  pod-template-hash: "6876878b9b"  # Injected automatically by Deployment!

This isolates the selector of each individual ReplicaSet, preventing race conditions during rolling deployments.

The spec.template Immutability Trap

A frequent Senior-level interview trap:

“What happens to currently running Pods if we run kubectl edit rs and update the container image from app:1.0 to app:2.0 directly inside a ReplicaSet?”

Answer: Absolutely nothing happens to the running Pods!

  • The spec.template in a ReplicaSet is evaluated only when creating new Pods (when actual < desired).
  • The ReplicaSet controller does NOT reconcile configuration drift of active Pods against the template; it monitors only the quantity of matching Pods.
  • Pods running image 1.0 will stay running indefinitely until they are manually deleted or crash. New Pods with image 2.0 will only spawn to replace missing replicas.

4 Tricky Questions with In-Depth Answers

1. If you modify spec.template.spec.containers[0].image directly in a running ReplicaSet, will currently running Pods be updated?

Answer: No, existing Pods continue running on the old image. The ReplicaSet controller is strictly a counter and lifecycle daemon, not a deployment engine. It queries the cluster to count the number of live Pods matching its matchLabels selector. Because the number of running Pods matches spec.replicas, the controller does not issue any mutation commands. The updated template is used only when a new Pod is scheduled (e.g., during scale-up or replacement). To achieve rolling updates across active Pods, you must use a Deployment.

2. What happens if two different ReplicaSet manifests have overlapping label selectors (e.g. app: myapp)?

Answer: A catastrophic Flapping Reconciliation Loop (Race Condition) occurs:

  • If both ReplicaSets have replicas: 2, and 4 Pods with app: myapp exist, each controller inspects the cluster, observes 4 matching Pods, decides that its target quota (2) is exceeded, and issues delete commands.
  • Once the Pod count drops below 2, both controllers simultaneously detect a shortage and issue create commands.
  • The two controllers fight indefinitely, continuously creating and destroying Pods, generating massive API server write contention and network churn. Kubernetes does not validate overlapping selectors at admission time; preventing selector conflicts is the engineer’s responsibility.

3. How do cascading deletion strategies (Foreground, Background, Orphan) differ when deleting a ReplicaSet?

Answer: When executing kubectl delete rs <name> --cascade=<policy>:

  • Background (Default): Kubernetes deletes the parent ReplicaSet object from etcd immediately. The Kubernetes garbage collector daemon subsequently cleans up child Pods asynchronously in the background.
  • Foreground: The parent ReplicaSet enters a deletionTimestamp state with the foregroundDeletion finalizer. It remains visible in etcd until all child Pods are completely terminated and removed. Only then is the ReplicaSet record deleted.
  • Orphan: The ReplicaSet is deleted from etcd, but child Pods have their ownerReferences detached. The Pods continue running indefinitely as orphaned bare Pods.

4. In what specific order does a ReplicaSet select which Pod to terminate during a Scale Down (from 5 to 2)?

Answer: The ReplicaSet controller uses a deterministic sorting algorithm to choose termination candidates:

  1. Unscheduled / Pending Pods: Deleted first.
  2. Unhealthy Pods: Pods in Unknown or Failed phases.
  3. Unready Pods: Pods where Ready=False (failed readiness probes).
  4. Node Colocation Balance: Pods on nodes hosting the highest concentration of replicas for this ReplicaSet (to preserve high availability and topological spread across the cluster).
  5. Age Tie-Breaker: If all Pods are equally healthy, the youngest Pod (most recent creationTimestamp) is terminated first. Older Pods are preserved because their JVM runtime and JIT compiler caches are already warmed up.

🎯 Interview Cheat Sheet

Summary of ReplicaSet

  • Core Function: Low-level controller ensuring exactly $N$ Pod replicas exist at all times.
  • Reconciliation: Compares spec.replicas against active matching Pods via matchLabels.
  • Relationship with Deployment: Deployment orchestrates ReplicaSets to achieve zero-downtime rolling updates and rollbacks.
  • Template Drift: Modifying spec.template in a running ReplicaSet does not update active Pods.
  • Selectors: Modern set-based matching (In, NotIn, Exists).
  • Cascade Deletion: --cascade=orphan allows Pods to outlive controller deletion.

Scale-Down Termination Priority List

1. Pending / CrashLoopBackOff Pods (Unscheduled)
2. Pods with Ready=False (Failing readiness probes)
3. Pods on nodes with the highest replica density (Cluster rebalancing)
4. Youngest Pods (Preserving JIT-warmed, older JVM instances)

Red Flags (DO NOT Say)

  • ❌ “I edit the container image in the ReplicaSet to update my microservice.” (Active Pods are never updated by a ReplicaSet; use Deployments).
  • ❌ “In production, we create raw kind: ReplicaSet manifests in our Helm charts.” (Anti-pattern; always declare Deployments).
  • ❌ “ReplicaSet kills pods at random when scaling down.” (It follows a strict 5-stage deterministic priority sequence).