🐳 Section 14 · Question #17

What is liveness probe

A Liveness Probe is a diagnostic mechanism in Kubernetes executed periodically by the node agent (kubelet) to determine whether an application inside a container is running prop...

🐳 Docker Kubernetes 📄 Source Markdown

🟢 Junior Level

30-Second Summary

A Liveness Probe is a diagnostic mechanism in Kubernetes executed periodically by the node agent (kubelet) to determine whether an application inside a container is running properly or has frozen into an unrecoverable state.

If a containerized process hangs (due to a Java thread deadlock, an infinite loop, or a corrupted state machine), the Linux operating system still considers the process ID active, even though it can no longer serve traffic:

  • kubelet sends a periodic probe request (HTTP, TCP, gRPC, or exec command).
  • If the application fails to respond or returns an error several times consecutively (failureThreshold), Kubernetes deems the container dead and forcibly terminates and restarts it (Self-Healing).

Real-World Analogy

A Liveness Probe acts like a heart rate monitor on an intensive care patient:

  • The attending nurse (kubelet) checks the patient’s pulse every 10 seconds.
  • As long as the heart beats steadily, no medical intervention is needed.
  • If the pulse stops for three consecutive checks, the medical team immediately applies a defibrillator (container restart) to shock the system back into functional operation.

Basic Manifest for Spring Boot 3 on Java 21

apiVersion: v1
kind: Pod
metadata:
  name: payment-app
spec:
  containers:
  - name: payment-service
    image: payment-service:1.0
    livenessProbe:
      httpGet:
        path: /actuator/health/liveness # Spring Boot Actuator liveness endpoint
        port: 8080
      initialDelaySeconds: 30 # Delay before first check to allow JVM startup
      periodSeconds: 10       # Probe interval (every 10 seconds)
      timeoutSeconds: 3       # Max duration to wait for response
      failureThreshold: 3     # Restart container after 3 consecutive failures

[!IMPORTANT] Primary Goal of Liveness: Detect situations where the application is completely deadlocked or corrupted and cannot self-recover without a full process restart.

🟡 Middle Level

The 4 Probe Execution Mechanisms

Mechanism Operational Mechanism Standard Use Case
httpGet Kubelet sends an HTTP GET request. Status codes 200–399 are considered successful. Web applications, REST APIs (Spring Boot Actuator).
tcpSocket Validates that a TCP socket connection can be established on the specified port. Databases, caches (PostgreSQL, Redis), message brokers.
grpc Invokes the official gRPC Health Checking Protocol (K8s 1.24+ GA). High-performance microservices communicating over gRPC.
exec Executes a shell command inside the container namespace (exit code 0 = success). Legacy systems, checking internal lockfiles (test -f /tmp/ok).

Cardinal Architectural Rule: Liveness vs. External Dependencies

[!CAUTION] NEVER CHECK EXTERNAL DEPENDENCIES (DATABASES, KAFKA, DOWNSTREAM APIS) IN A LIVENESS PROBE!

The Cascading Death Spiral Scenario:

  1. Your production PostgreSQL database experiences a temporary latency spike or connection pool saturation for 15 seconds.
  2. If your 50 Java backend pods evaluate database health inside their liveness probes, all 50 probes simultaneously fail.
  3. Kubelet across the cluster kills and restarts all 50 pods at once!
  4. As all 50 pods restart simultaneously, their HikariCP connection pools hammer the recovering database with thousands of concurrent connection requests, crashing the database permanently.
    • Rule: A Liveness Probe must evaluate STRICTLY internal container health (memory, thread deadlocks, local event loops). External dependencies must be delegated solely to the Readiness Probe.

Native Support in Spring Boot Actuator

Spring Boot 2.3+ and Spring Boot 3+ automatically integrate with Kubernetes by exposing dedicated health group probes:

  • http://localhost:8080/actuator/health/liveness — Exposes internal container health (LivenessState). Excludes all remote database, cache, and broker checks.
  • http://localhost:8080/actuator/health/readiness — Checks connection pool readiness, database connectivity, and messaging queues.

Enable both probes in application.yaml:

management:
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true
  endpoint:
    health:
      probes:
        enabled: true

🔴 Senior Level

The Tandem of startupProbe and livenessProbe

A classic operational challenge with enterprise Java applications is slow cold startup (Spring Boot context scanning, Liquibase/Flyway migrations, and JPA metamodel generation often take 40–90 seconds):

  • If you configure initialDelaySeconds: 15, the Liveness Probe kills a healthy JVM mid-initialization, triggering a boot-loop.
  • If you configure initialDelaySeconds: 90, when a deadlock occurs in production during steady state, Kubelet waits 90 seconds before taking initial recovery action.

The Solution — startupProbe:

containers:
- name: payment-service
  image: payment-service:1.0
  startupProbe:
    httpGet:
      path: /actuator/health/liveness
      port: 8080
    periodSeconds: 5
    failureThreshold: 30 # Grants 30 * 5s = 150 seconds for cold initialization!

  livenessProbe:
    httpGet:
      path: /actuator/health/liveness
      port: 8080
    periodSeconds: 10
    failureThreshold: 3  # Fast 30-second response window to runtime deadlocks

Until startupProbe succeeds, both livenessProbe and readinessProbe remain completely disabled!

HotSpot JVM Internals: False Positives During STW Full GC Pauses

When the JVM garbage collector (e.g., Parallel GC, or G1 experiencing memory fragmentation) triggers a heavy Stop-The-World (STW) Full GC:

  1. All application execution threads (including embedded Tomcat/Netty request acceptor and worker threads) are paused by the JVM safepoint mechanism.
  2. A probe packet dispatched by kubelet arrives at the container network interface, but Tomcat cannot construct or return an HTTP response.
  3. If timeoutSeconds: 1 is configured aggressively, Kubelet records a probe failure. Three consecutive STW pauses will prompt Kubelet to terminate a healthy JVM simply busy reclaiming memory. Recommendation: Set timeoutSeconds: 3 to 5 and failureThreshold: 3 to provide adequate tolerance for temporary GC spikes.

Programmatic State Invalidation in Spring Boot 3

Java code can programmatically signal Kubernetes to trigger a container restart upon discovering unrecoverable background corruption:

@Service
public class CriticalWorkerService {

    private final ApplicationEventPublisher eventPublisher;

    public CriticalWorkerService(ApplicationEventPublisher eventPublisher) {
        this.eventPublisher = eventPublisher;
    }

    public void processCriticalQueue() {
        try {
            // Process mission-critical background queue
        } catch (FatalCorruptionException ex) {
            // Transition LivenessState to BROKEN -> Actuator returns HTTP 503
            // Kubelet registers probe failure and orchestrates container restart
            AvailabilityChangeEvent.publish(eventPublisher, this, LivenessState.BROKEN);
        }
    }
}

4 Tricky Questions

1. What happens if a container fails its Liveness Probe, but the Pod manifest specifies restartPolicy: Never?

Answer: Kubelet will NOT restart the container. Under Kubernetes pod lifecycle rules, if restartPolicy: Never is set:

  • Kubelet forcefully stops the unresponsive container by sending SIGTERM, followed by SIGKILL.
  • However, because restarts are explicitly disabled, Kubelet does not attempt to recreate the container.
  • The container transitions into Terminated status, and the entire Pod permanently transitions into the terminal Failed phase until manually deleted or garbage-collected by the controller.

2. How can you detect a multithreaded Java Deadlock in production using a Liveness Probe?

Answer: By registering a custom Spring Boot Actuator HealthIndicator utilizing the JVM’s ThreadMXBean:

@Component
public class DeadlockHealthIndicator implements HealthIndicator {
    private final ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();

    @Override
    public Health health() {
        long[] deadlocked = threadBean.findDeadlockedThreads();
        if (deadlocked != null && deadlocked.length > 0) {
            return Health.down()
                         .withDetail("deadlockedThreadsCount", deadlocked.length)
                         .build();
        }
        return Health.up().build();
    }
}

If two or more threads enter a circular monitor wait deadlock, the health indicator immediately reports DOWN (HTTP 503). Kubelet detects the failure and restarts the JVM container, resolving the deadlock without human intervention.

3. What is the fundamental difference in Kubernetes action when a livenessProbe fails versus when a readinessProbe fails?

Answer:

  • livenessProbe Failure: Kubernetes executes a destructive action — it terminates and restarts the container. The Pod IP remains unchanged, but inflight in-memory state is erased, and restart counts increment.
  • readinessProbe Failure: Kubernetes executes a non-destructive routing action — it does NOT restart the container. Instead, it dynamically removes the Pod’s IP address from the Service Endpoints / EndpointSlice. The Pod remains running and can continue executing background tasks or completing existing requests, but no new external client traffic will be routed to it until the probe returns HTTP 200 again.

4. Why is executing Liveness Probes via the exec mechanism (e.g., command: ["curl", "localhost:8080/health"]) considered an anti-pattern in high-throughput workloads?

Answer: When using exec, Kubelet instructs the container runtime (containerd / CRI-O) to fork and spawn a brand-new operating system process inside the container namespace for every health check interval:

  • Spawning OS processes (fork + exec) consumes substantial CPU and memory descriptors and triggers kernel context switches, especially when repeated across dozens of containers every 5 seconds.
  • Modern security best practices utilize Distroless or minimal Alpine base images that do not contain curl, wget, or shell interpreters (sh/bash).
  • Native httpGet probes open a direct TCP socket connection from Kubelet to the container port over the host network bridge with zero process-spawn overhead.

🎯 Interview Cheat Sheet

Core Liveness Probe Architecture

  • Primary Function: Verifies container process health; triggers a container restart on repeated failures.
  • Golden Invariant: Never evaluate external databases, caches, or third-party APIs in a Liveness Probe.
  • Startup Probe Coupling: Pair slow-booting JVM apps with startupProbe to avoid false initial restart loops.
  • Spring Boot 3 Actuator: Always leverage the dedicated /actuator/health/liveness endpoint.
  • GC STW Tolerance: Ensure timeoutSeconds (3–5s) exceeds typical Full GC pauses to avoid false restarts.
  • Liveness vs. Readiness: Liveness restarts the container; Readiness removes the Pod from Service load balancers.

Comparison of the 3 Probe Types

| Probe Type | Execution Window | Action on Failure | Core Objective | | :— | :— | :— | :— | | startupProbe | During container boot only | Terminates and restarts container | Grants long grace window for JVM classloading | | livenessProbe | After startup succeeds, continuous | Restarts container | Unfreezes deadlocks and infinite loops | | readinessProbe | After startup succeeds, continuous | Removes Pod from Service Endpoints | Protects against traffic surges & warm-up latency |

Red Flags (What NOT to Say)

  • ❌ “Our Liveness Probe connects to Postgres and Kafka to verify cluster connectivity.” — Guarantees a cluster-wide Cascading Death Spiral during database failovers.
  • ❌ “Setting initialDelaySeconds: 5 is best practice for Spring Boot microservices.” — The JVM will not finish loading before Kubelet kills it.
  • ❌ “Liveness and Readiness probes can both point to /actuator/health without parameters.” — Default Actuator /health includes database checks, violating the isolation rule.