What is readiness probe

A Readiness Probe is a Kubernetes diagnostic mechanism executed continuously by the node agent (kubelet) to determine whether an application Pod is ready to accept and process i...

🟢 Junior Level

30-Second Summary

A Readiness Probe is a Kubernetes diagnostic mechanism executed continuously by the node agent (kubelet) to determine whether an application Pod is ready to accept and process incoming client network traffic.

Unlike a Liveness Probe, a Readiness Probe NEVER terminates or restarts the container:

If a readiness check fails, Kubernetes takes a single non-destructive action: it temporarily removes the Pod’s IP address from the Service load-balancing pool (Endpoints / EndpointSlice).
Client traffic is diverted away from the unready Pod and distributed across healthy peer replicas.
As soon as the container finishes warming up caches, establishing database pools, or clearing memory buffers, the probe succeeds, and Kubernetes automatically rejoins the Pod to the Service routing pool.

Real-World Analogy

A Readiness Probe is like a “10-MINUTE BREAK” sign placed on a supermarket checkout counter:

The cashier is alive, well, and not fired (the container is NOT restarted).
However, the cashier is currently reloading the receipt tape or counting cash register drawers (warming up caches, establishing DB connections).
Customers in line are directed to adjacent open checkout lanes (traffic is routed to other pods).
Once the cashier finishes restocking, the sign is removed, and customer checkout resumes immediately.

Basic Manifest for Spring Boot 3

apiVersion: v1
kind: Pod
metadata:
  name: order-service
spec:
  containers:
  - name: order-app
    image: order-service:1.0
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness # Spring Boot Actuator readiness endpoint
        port: 8080
      initialDelaySeconds: 15 # Delay before first check
      periodSeconds: 5        # Check frequency (every 5 seconds)
      failureThreshold: 2     # Evict from endpoints after 2 consecutive failures
      successThreshold: 1     # Re-admit to traffic after 1 successful check

In the output of kubectl get pods, readiness status is reflected directly in the READY column:

1/1 Running: The container process is active and actively serving client traffic.
0/1 Running: The container process is active, but traffic is blocked due to a failing readiness probe.

🟡 Middle Level

The Role of Readiness Probes in Zero-Downtime Rolling Updates

Without a configured Readiness Probe, true Zero-Downtime Deployment is impossible:

You deploy a new release v2.
The Linux kernel starts the container process, assigns a PID, and Kubernetes flags the Pod status as Running.
If no Readiness Probe is defined, Kubernetes instantly starts routing user requests to the new Pod on second one, before the JVM has even loaded the Spring ApplicationContext!
End users experience an immediate surge of 502 Bad Gateway and Connection Refused errors.
With a Readiness Probe: Kubernetes suppresses traffic until the application explicitly responds with HTTP 200 on /actuator/health/readiness, ensuring a seamless, error-free transition from v1 to v2.

Side-by-Side Comparison: Liveness vs. Readiness

Attribute	Liveness Probe	Readiness Probe
Action on Failure	Forcibly terminates and restarts container (`SIGKILL`)	Removes Pod IP from `Endpoints` (traffic isolation)
Container Lifecycle	Container restarts; in-memory state is wiped	Container keeps running; continues background processing
What to Inspect	Internal process lifecycle, deadlocks, fatal thread halts	Liquidity of connection pools, cache hydration, queues
Deployment Impact	Does not coordinate rolling update traffic switch	Blocks rolling deployment until new replica is fully ready

Spring Boot 3 `ReadinessState` in Application Code

Spring Boot 3 manages readiness via the internal ReadinessState event bus:

ReadinessState.ACCEPTING_TRAFFIC: Context is initialized, connection pools are live. Actuator returns HTTP 200 OK.
ReadinessState.REFUSING_TRAFFIC: The service is temporarily unavailable. Actuator returns HTTP 503 Service Unavailable.

Engineers can programmatically detach a Pod from the load balancer during resource-intensive operations (e.g., priming an in-memory cache):

@Service
public class CacheWarmupService {

    private final ApplicationEventPublisher eventPublisher;

    public CacheWarmupService(ApplicationEventPublisher eventPublisher) {
        this.eventPublisher = eventPublisher;
    }

    public void reloadLargeProductCatalog() {
        // 1. Temporarily withdraw this Pod from the Kubernetes load balancer
        AvailabilityChangeEvent.publish(eventPublisher, this, ReadinessState.REFUSING_TRAFFIC);

        try {
            // 2. Perform memory-intensive database hydration
            loadCatalogIntoMemory();
        } finally {
            // 3. Re-enable traffic routing once warm-up completes
            AvailabilityChangeEvent.publish(eventPublisher, this, ReadinessState.ACCEPTING_TRAFFIC);
        }
    }
}

🔴 Senior Level

Network Propagation Lag & The PreStop Hook Tandem

A critical insight for senior engineers is that removing a Pod from Kubernetes routing is asynchronous and takes time:

Kubelet observes probe failure and updates kube-apiserver (~1s).
The EndpointSlice Controller updates the EndpointSlice object (~1s).
The kube-proxy daemon on every node recalculates and rewrites local iptables / IPVS rules (~1–3s).
The cumulative propagation delay ranges from 2 to 5 seconds! During these 3–5 seconds, incoming client packets are still routed to the Pod’s local socket.

Architectural Mitigation: Readiness + PreStop Hook

To ensure 100% zero-packet-drop during pod shutdowns or rolling deployments:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"] # 10-second grace window before SIGTERM

During this 10-second pause, kube-proxy finishes updating iptables across all cluster nodes. Client traffic cleanly subsides before Spring Boot receives SIGTERM and initiates graceful shutdown.

Cloud Readiness Gates

In modern cloud managed environments (e.g., AWS EKS with AWS Load Balancer Controller or Google Cloud GKE NEG):

Kubelet verifies the local container port and marks the Pod ready (PodReady=True).
However, the external AWS Application Load Balancer (ALB) has not yet registered the Pod’s target IP in its target group or passed its own HTTP health checks.
If Kubernetes shifts traffic immediately, external clients accessing via Ingress encounter connection dropouts.

Readiness Gates solve this cloud synchronization race:

spec:
  readinessGates:
  - conditionType: "target-health.elbv2.k8s.aws/v2" # External cloud gate

Kubernetes will not treat the Pod as Ready until the external cloud controller (AWS ALB) confirms successful target registration and external probe passage!

The Database Check Anti-Pattern in Readiness Probes

A prevalent high-load architecture trap:

If 200 microservice replicas include a live PostgreSQL query in their Readiness Probes, a transient database lock or slow query will cause all 200 pods to simultaneously fail readiness checks.
Kubernetes immediately empties the Service Endpoints object to zero entries.
The entire system suffers a total outage (503 Service Unavailable) for 100% of incoming users, even for HTTP endpoints that do not require database access. Best Practice: Keep Readiness Probes focused on local container capacity (thread pool saturation, local memory). Handle external database latency via client-side Circuit Breakers (e.g., Resilience4j) with graceful fallback responses.

4 Tricky Questions

1. Why is a `readinessProbe` alone insufficient to guarantee 100% Zero-Downtime deployments, and why is a `preStop` hook containing `sleep 10` strictly required?

Answer: Pod eviction from routing and container shutdown occur concurrently and independently:

When a Pod termination event occurs, Kubelet immediately sends a SIGTERM signal to the container process. Spring Boot starts shutting down its web server and stops accepting incoming TCP connections.
Simultaneously, kube-apiserver notifies the Endpoint controller, which instructs kube-proxy on all worker nodes to remove the Pod IP from their iptables / IPVS tables.
This cluster-wide iptables synchronization takes between 1 and 4 seconds.
During this propagation window, clients continue dispatching HTTP requests to the terminating Pod, receiving Connection Refused or 502 Bad Gateway errors because the socket is already closed.
A preStop: exec: command: ["sh", "-c", "sleep 10"] hook delays the delivery of SIGTERM by 10 seconds, guaranteeing that all cluster nodes update their packet-routing rules before the application begins its shutdown sequence.

2. What happens to network traffic and Pod state if `readinessProbe` returns HTTP 500 while `livenessProbe` returns HTTP 200 simultaneously?

Answer:

Container State: The container is NOT restarted. It remains running in memory. In kubectl get pods, the status remains Running, but the READY count changes from 1/1 to 0/1.
Network Traffic: Kubernetes removes the Pod’s IP address from the Endpoints and EndpointSlice objects of all linked Services.
Outcome: No new external traffic arriving via ClusterIP, NodePort, or Ingress will reach this container. However, the container can still send outbound requests to databases, consume messages from Kafka, and complete in-flight transactions until the readiness probe returns HTTP 200 again.

3. What is the purpose of `successThreshold` in a Readiness Probe configuration, and why is it set to 1 by default?

Answer:

successThreshold defines the minimum number of consecutive successful probe executions required before Kubernetes marks an unready container as Ready and re-admits it to the Service load balancer.
The default value is 1 to minimize delay when bringing a newly booted, healthy Pod into production rotation.
In workloads with erratic startup performance or heavy cold-start JIT warming, increasing successThreshold: 2 or 3 prevents Kubernetes from routing client traffic to a Pod that intermittently returns a single success before stabilizing.

4. How does a failing Readiness Probe affect a Kubernetes Headless Service (`clusterIP: None`)?

Answer: It directly affects DNS resolution. For a Headless Service, CoreDNS creates DNS A-records by scraping the associated Endpoints object:

If a Pod fails its readinessProbe, its IP is removed from subsets.addresses and moved to subsets.notReadyAddresses.
By default, CoreDNS stops returning the failing Pod’s IP address in DNS queries.
However, if the Headless Service manifest explicitly sets publishNotReadyAddresses: true, CoreDNS continues returning all Pod IPs regardless of readiness state. This is essential for stateful consensus clusters (e.g., Elasticsearch, Cassandra, or etcd) that require peer discovery to replicate state before individual nodes become ready to serve clients.

🎯 Interview Cheat Sheet

Core Readiness Architecture

Primary Function: Verifies container readiness to handle user traffic; failures isolate the Pod from Service routing without restarting the process.
Zero-Downtime Requirement: Mandatory for rolling deployments; prevents traffic from reaching uninitialized JVM instances.
PreStop Hook Coupling: Pair with preStop: sleep 10 to bridge the 2–4 second kube-proxy iptables propagation lag.
Spring Boot 3 Actuator: Exposes /actuator/health/readiness bound to the internal ReadinessState state machine.
Readiness Gates: Bridges external cloud load balancer health checks (AWS ALB Target Groups) with Kubernetes pod readiness.
Database Decoupling: Never make global DB connectivity a hard readiness probe condition across all replicas; use Circuit Breakers instead.

Health Probes Comparison

Red Flags (What NOT to Say)

❌ “If the Readiness Probe fails 3 times, Kubernetes restarts the pod.” — Fatal misconception; only Liveness and Startup probes trigger restarts.
❌ “A Pod with status Running is immediately ready to receive traffic without a Readiness Probe.” — Running only means the container process was spawned; JVM initialization can take another 45 seconds.
❌ “We include full database integration tests in our Readiness Probe.” — A minor DB slowdown will knock 100% of your pods out of traffic simultaneously.

What is liveness probe — Unfreezing stuck processes
Why are health checks needed — Health check strategy and telemetry
How to organize rolling update in Kubernetes — Zero-downtime rollouts
What is Service in Kubernetes — Service routing and EndpointSlices