🐳 Section 14 · Question #24

How to monitor applications in Kubernetes

Observability is like the instrument flight deck and flight data recorder of a modern commercial airliner:

🐳 Docker Kubernetes 📄 Source Markdown

🟢 Junior Level

30-Second Summary

Comprehensive Observability in Kubernetes is anchored on the “Three Pillars of Observability”:

  1. Metrics (What is happening, and when, quantified in numbers?): Numerical time-series telemetry (CPU/memory utilization, request throughput RPS, p99 latency, HTTP 5xx error rates). The industry standard stack is Prometheus + Grafana.
  2. Logs (Why did it happen?): Textual event streams emitted to container stdout/stderr. In Kubernetes, log agents (Promtail, Fluent Bit, Vector) deployed as DaemonSets scrape and ship logs to centralized datastores (Grafana Loki, Elasticsearch / OpenSearch).
  3. Distributed Tracing (Where was the request delayed?): End-to-end request path propagation across distributed microservice hops with granular span timings (OpenTelemetry, Jaeger, Grafana Tempo).

Real-World Analogy

Observability is like the instrument flight deck and flight data recorder of a modern commercial airliner:

  • Metrics: The speedometer, altimeter, engine tachometer, and fuel gauges. The flight crew views aggregate aircraft status at a glance on electronic screens (Grafana).
  • Logs: The flight data and cockpit voice recorder (“Black Box”), storing a timestamped transcript of every crew command and system alarm.
  • Tracing: An airline passenger luggage tracking tag: check-in $\to$ conveyor sorting $\to$ cargo hold loading $\to$ transit transfer $\to$ baggage claim (pinpointing the precise transit airport where a bag encountered delays).

Quick Start: Spring Boot 3 + Micrometer + Prometheus

In modern enterprise Java applications, telemetry exposure requires two dependencies:

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yaml
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus # Expose /actuator/prometheus endpoint
  metrics:
    export:
      prometheus:
        enabled: true

🟡 Middle Level

Architecture of kube-prometheus-stack (The Pull Model)

[ Physical Worker Nodes ]     [ Kubelet / Cgroups ]     [ K8s API Server ]      [ Java Spring Boot Pods ]
           │                           │                         │                          │
     Node Exporter                  cAdvisor            kube-state-metrics             Micrometer
 (OS, CPU, RAM, Disk I/O)    (Container CPU/Memory)    (Pod phases, crash counts)   (/actuator/prometheus)
           ▲                           ▲                         ▲                          ▲
           └───────────────────────────┼─────────────────────────┼──────────────────────────┘
                                       │ (HTTP GET Scrape every 15–30 seconds)
                             [ Prometheus Server ] ──► Time-Series Database (TSDB)
                                       │
                      ┌────────────────┴────────────────┐
                      ▼                                 ▼
            [ Grafana Dashboards ]            [ Alertmanager (Slack / PagerDuty) ]
  • Prometheus Pull Model: Prometheus actively polls (scrapes) discovered HTTP metric endpoints every 15–30 seconds via dynamic Kubernetes Service Discovery.
  • Specialized Metric Collectors:
    • Node Exporter (DaemonSet): Host hardware metrics (physical memory, network bandwidth, disk I/O).
    • cAdvisor (built into kubelet): Container-level resource consumption directly from Linux cgroups.
    • kube-state-metrics (Deployment): Kubernetes object states (unready replicas, container restart counts, PVC storage capacities).

Declarative Monitoring via ServiceMonitor (Prometheus Operator)

Modern Kubernetes clusters eliminate manual prometheus.yml edits by defining declarative ServiceMonitor Custom Resources:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-monitor
  namespace: billing
  labels:
    release: prometheus # Bound to Prometheus instance selector
spec:
  selector:
    matchLabels:
      app: payment-service
  endpoints:
  - port: http-metrics
    path: /actuator/prometheus
    interval: 15s

Top 5 PromQL Queries for Java Engineers

# 1. JVM Heap Memory Utilization Percentage:
(jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}) * 100

# 2. JVM Garbage Collection Pause Duration Rate (Stop-The-World Time):
rate(jvm_gc_pause_seconds_sum[5m])

# 3. HTTP Request Throughput per URI (RPS):
sum(rate(http_server_requests_seconds_count[1m])) by (uri)

# 4. HTTP 5xx Error Rate Percentage:
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/ sum(rate(http_server_requests_seconds_count[5m])) * 100

# 5. API Response Time 99th Percentile (Latency p99):
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le))

🔴 Senior Level

The Cardinality Explosion Catastrophe

The most frequent cause of Prometheus production outages in high-throughput microservices is High Cardinality:

  • Theory: In Prometheus TSDB, every unique combination of label=value pairs provisions a completely separate, dedicated time-series stream in memory and on disk.
  • The Fatal Developer Anti-Pattern: 
    // ❌ DISASTROUS ANTI-PATTERN: Emitting unique transaction/user identifiers as metric labels
meterRegistry.counter("orders.created",
    "userId", order.getUserId(),      // 10,000,000 unique values
    "orderId", order.getOrderId()     // 50,000,000 unique values
).increment();
  • Consequences: Within hours, the time-series count surges from 200,000 to 50,000,000. Prometheus memory usage spirals, triggering OOMKilled (Exit Code 137). Upon restarting, the daemon hangs indefinitely attempting to replay its bloated Write-Ahead Log (WAL).
  • Architectural Rule: Metric labels must adhere to Low Cardinality: fixed HTTP methods (GET/POST), status groups (200/500), or error categories (TimeoutException/ValidationError). Dynamic entity identifiers (userId, orderId) must be emitted strictly to Application Logs or Tracing Spans!

Long-Term Retention and Scaling: Thanos and VictoriaMetrics

Prometheus TSDB is architecturally optimized for local, short-term SSD storage (15–30 days) and lacks native horizontal clustering. Enterprise environments scale Prometheus horizontally using distributed extensions:

  • Thanos: Deploys a thanos-sidecar adjacent to each Prometheus pod. The sidecar uploads historical TSDB metric blocks to inexpensive object storage (AWS S3, Google Cloud Storage, Ceph). Thanos Querier federates multi-cluster queries over 3–5 years of data with automated downsampling (5m/1h resolution rollups).
  • VictoriaMetrics: A high-performance, cost-efficient drop-in replacement for Prometheus TSDB that provides native PromQL support, 4–7x lower RAM consumption, and transparent long-term storage clustering.

Managing OpenTelemetry Tracing Overhead in High-Throughput Systems

Tracing every transaction in a 50,000 RPS microservice topology produces immense CPU serialization overhead and floods cluster networks with terabytes of telemetry.

The Solution — Tail-Based Sampling:

  • Microservices forward 100% of raw spans to an OpenTelemetry Collector cluster via OTLP.
  • The collector temporarily buffers full trace lifecycles in memory.
  • If a trace finishes with HTTP 200 OK in 15 ms, the collector discards it (drops standard traffic).
  • If any span inside the trace records HTTP 500 or exceeds a 1-second latency threshold, the collector persists the complete trace (100% anomaly capture).

4 Tricky Questions

1. Why does Prometheus rely on a Pull model by default, and how do you monitor short-lived batch jobs (CronJobs, Serverless functions) that finish before the scrape interval?

Answer:

  • Why Pull: The pull model allows the monitoring server to control metric ingestion rates (Rate Limiting), preventing monitoring infrastructure from being overwhelmed during application traffic spikes. Furthermore, if an application stops responding to scrape polls, Prometheus immediately flags up == 0, providing instant out-of-the-box health checks.
  • Handling Short-Lived Jobs: If a batch container launches, completes execution in 3 seconds, and terminates, a 15-second scrape loop will never discover or scrape it.
  • Solution — Prometheus Pushgateway:
    1. The short-lived batch job pushes its calculated metrics to an intermediary Pushgateway service via HTTP POST prior to process exit.
    2. The Pushgateway retains the metrics in memory.
    3. Prometheus scrapes the Pushgateway on its normal 15-second pull schedule.

2. What is the fundamental architectural difference between cAdvisor and kube-state-metrics, and which tool reveals why a Pod crashed with CrashLoopBackOff?

Answer:

  • cAdvisor (Container Advisor): Gathers physical runtime resource usage telemetry from Linux kernel cgroups (actual CPU nanoseconds, resident memory RSS bytes, network interface packets, disk I/O). It has no awareness of Kubernetes manifests, labels, or controller states.
  • kube-state-metrics: Watches the kube-apiserver via informers and transforms the declarative state of Kubernetes objects into Prometheus metrics.
  • Only kube-state-metrics can expose why a Pod is failing, emitting metrics such as kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} alongside kube_pod_container_status_last_terminated_reason{exit_code="137"}. cAdvisor cannot provide the exit status or crash reason.

3. How can HotSpot JVM Garbage Collection Stop-The-World (STW) pauses distort in-application latency metrics, and how is this discrepancy detected?

Answer: When an application computes latency using Micrometer code annotations (@Timed) or filter timers:

  • During a Full GC pause, all application threads are suspended at JVM safepoints, including the internal application timer threads.
  • If an incoming HTTP packet reaches the node network interface during a 2-second STW pause, the client experiences a 2010 ms response time. However, the in-application filter only begins timing after the GC pause clears, falsely recording a “clean” execution time of 10 ms!
  • Detection: Correlate internal application latency metrics against external edge measurements recorded by the Ingress Controller (NGINX $request_time) or a Service Mesh (Envoy). A large delta between Ingress Latency and Application Latency indicates severe GC STW pauses or TCP socket connection queue saturation.

4. How is distributed trace context propagated across asynchronous boundaries in Java (CompletableFuture, Spring WebFlux, Virtual Threads)?

Answer: Legacy tracing frameworks relied on ThreadLocal storage to hold the active TraceId and SpanId:

  • When switching execution to asynchronous worker pools (CompletableFuture.supplyAsync, Project Reactor / WebFlux), the operating system thread changes, causing ThreadLocal context to be lost, breaking distributed traces into detached fragments.
  • Modern Architectural Solutions:
    1. Reactive Programming (Spring WebFlux): Context is propagated via Reactor’s immutable subscriber context (reactor.util.context.Context), flowing upstream through operators.
    2. OpenTelemetry Java Agent: Bytecode instrumentation (-javaagent:opentelemetry-javaagent.jar) intercepts Executor, ForkJoinPool, and Callable factories, automatically capturing and attaching active Context instances across thread transitions.
    3. Java 21 Virtual Threads: High-scale virtual threads preserve execution boundaries cleanly using Java 21 Scoped Values (JEP 446 / JEP 481) or optimized inheritable thread contexts.

🎯 Interview Cheat Sheet

Core Kubernetes Monitoring Principles

  • Three Pillars of Observability: Metrics (Prometheus), Logs (Loki / ELK), Traces (OpenTelemetry / Jaeger).
  • Prometheus Standard: Pull-based metric polling (every 15–30s) targeting /actuator/prometheus.
  • Infrastructure Metric Architecture:
    • Node Exporter $\to$ Physical node hardware stats.
    • cAdvisor $\to$ Container cgroup resource consumption.
    • kube-state-metrics $\to$ Kubernetes API object lifecycle states and restart codes.
  • Top Enemy — Cardinality Explosion: Never write dynamic IDs (userId, orderId, UUIDs) to Prometheus metric labels.
  • Long-Term Scaling: Use Thanos (S3 storage offload + downsampling) or VictoriaMetrics for multi-year telemetry retention.
  • SRE Golden Signals: Latency (p95/p99), Traffic (RPS), Errors (HTTP 5xx), Saturation (CPU/RAM).

Observability Tooling by Domain

| Telemetry Type | Standard Tools | Transport / Protocol | Primary Purpose | | :— | :— | :— | :— | | Metrics | Prometheus, VictoriaMetrics | PromQL / HTTP Pull | Real-time dashboards, alerting, capacity planning | | Logs | Grafana Loki, Fluent Bit, Vector | LogQL / HTTP Push | Error debugging, exception stack traces, auditing | | Traces | OpenTelemetry, Jaeger, Tempo | OTLP / W3C TraceContext | Pinpointing latency bottlenecks in microservice call chains |

Red Flags (What NOT to Say)

  • ❌ “We include user_id as a label in Prometheus to track individual user activity.” — Guarantees a Cardinality Explosion and crashes the monitoring server.
  • ❌ “Prometheus gathers application error logs from stdout.” — Prometheus processes time-series numbers only; log shipping requires Loki or ELK.
  • ❌ “We retain 3 years of raw metrics in standard Prometheus TSDB on the worker node SSD.” — Standard TSDB is not horizontally scalable; multi-year retention requires Thanos or VictoriaMetrics.