How to monitor a distributed microservices system

Google's Site Reliability Engineering framework defines the four critical signals for monitoring customer-facing applications: 4. Saturation: A measure of the most constrained s...

🟢 Junior Level

Observability and Monitoring in a distributed microservices architecture is the engineering practice and tooling suite that enables teams to infer the internal state, performance, and health of a complex system based on its external telemetry outputs: Metrics, Logs, and Traces.

                  ┌──────────────────────────────────────────────┐
                  │    The Three Pillars of Observability        │
                  └──────┬───────────────┬───────────────┬───────┘
                         │               │               │
                         ▼               ▼               ▼
                   [ METRICS ]       [ LOGS ]      [ TRACES ]
                  (Prometheus)      (Loki / ELK)   (Jaeger / OTel)
                  "What broke and   "Why did it     "Where in the call
                   when did it?"     happen?"        graph is the delay?"

The Google SRE Four Golden Signals

Google’s Site Reliability Engineering framework defines the four critical signals for monitoring customer-facing applications:

Latency: The time required to process a request (tracked separately for successful requests and failed requests).
Traffic: A measure of incoming demand on the system (e.g., Requests Per Second — RPS, or Transactions Per Second — TPS).
Errors: The rate of requests that fail explicitly (HTTP 5xx server errors, uncaught runtime exceptions, or timeout events).
Saturation: A measure of the most constrained system resource (e.g., JVM heap memory utilization, CPU core saturation, HikariCP database connection pool availability, or disk I/O queue depth).

🟡 Middle Level

1. Monitoring Methodologies: RED vs USE

Enterprise observability categorizes telemetry using two complementary methodologies:

Methodology	Target Scope	Core Telemetry Signals
RED Method	Request-driven applications and APIs (Microservices, HTTP/gRPC endpoints).	• Rate: Number of incoming requests per second. • Errors: Number of failed requests per second. • Duration: The distribution of request execution times (percentiles).
USE Method	Hardware and system infrastructure (CPUs, RAM, Network interfaces, Disk storage).	• Utilization: Percentage of time the resource was busy performing work. • Saturation: Volume of backlogged work queued waiting for the resource. • Errors: Count of physical hardware or packet drop errors.

2. Application Instrumentation: Spring Boot 3 Actuator + Micrometer

In the Java ecosystem, Micrometer provides a vendor-neutral metrics facade (similar to SLF4J for logging) that formats and exports metrics to TSDBs like Prometheus:

@RestController
@RequiredArgsConstructor
public class CheckoutController {

    private final MeterRegistry registry;
    private final OrderService orderService;

    private Counter checkoutSuccessCounter;
    private Counter checkoutFailCounter;
    private Timer checkoutTimer;

    @PostConstruct
    void initMetrics() {
        this.checkoutSuccessCounter = Counter.builder("checkout_orders_total")
            .description("Total successfully placed orders")
            .tag("status", "success")
            .register(registry);

        this.checkoutFailCounter = Counter.builder("checkout_orders_total")
            .description("Total failed checkout attempts")
            .tag("status", "failure")
            .register(registry);

        this.checkoutTimer = Timer.builder("checkout_duration_seconds")
            .description("Checkout execution latency histogram")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry);
    }

    @PostMapping("/checkout")
    public OrderResponse checkout(@RequestBody OrderRequest req) {
        return checkoutTimer.record(() -> {
            try {
                OrderResponse response = orderService.process(req);
                checkoutSuccessCounter.increment();
                return response;
            } catch (Exception ex) {
                checkoutFailCounter.increment();
                throw ex;
            }
        });
    }
}

3. Production PromQL Queries for Alerting

Calculating the HTTP 5xx error percentage over a 5-minute window:

sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m])) * 100 > 5

Calculating the 99th percentile (p99) latency across all cluster nodes:

histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le))

🔴 Senior Level

1. The Metric Cardinality Explosion Disaster

The single most dangerous failure mode in Time-Series Databases (TSDBs like Prometheus, VictoriaMetrics, and Mimir) is Cardinality Explosion:

Every distinct combination of a metric name and its label key-value pairs instantiates a new Time Series, which is held in in-memory indexing structures.

THE FATAL ANTI-PATTERN:

// PRODUCTION CATASTROPHE! High-cardinality user/order IDs in Prometheus labels!
registry.counter("order_created", "user_id", req.getUserId()).increment();

The Result: When 1,000,000 customers place orders, Prometheus generates 1,000,000 unique time-series entries. Memory consumption explodes exponentially, triggering a fatal OutOfMemoryError that crashes the entire monitoring cluster and blinds the SRE team during an incident.
Architectural Rule: Metric labels must strictly be Low-Cardinality (finite enumerations like http_status=200, region=us-east, payment_method=CARD). High-cardinality identifiers (userId, orderId, traceId) belong exclusively in structured logs and distributed traces.

2. SLI, SLO, SLA, and the Error Budget Framework

SLA (Legal Contract):           99.0% availability (financial penalties if breached)
             ▲
             │ (Safety Margin)
SLO (Internal Target):          99.9% availability
             ▲
             │ (Telemetry Measurement)
SLI (Indicator Metric):         (Successful Requests / Total Requests) over 30 days

Error Budget: The permissible fraction of unreliability. For an SLO of $99.9\%$, the Error Budget is $100\% - 99.9\% = 0.1\%$ of total monthly traffic.
The Error Budget Policy:
- While the system operates within its error budget, product teams have the green light to deploy risky features, run experiments, and iterate rapidly.
- If a production outage consumes the monthly error budget within the first week, feature deployments are frozen immediately. Engineering resources pivot 100% to stability, architecture refactoring, and automated testing until reliability recovers.

3. Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)

Traditional alerts (“Notify if error rate > 1% for 5 minutes”) suffer from two critical flaws:

They trigger false pages on short-lived transient blips, causing severe Alert Fatigue.
They miss slow, catastrophic drains (e.g., an error rate of 0.3% that silently burns through the entire monthly error budget over 3 days).

Google SRE Multi-Burn-Rate Architecture: Monitor the rate of error budget consumption (Burn Rate) across overlapping short and long windows:

14.4x Burn Rate over 1 hour: 2% of the monthly error budget will be consumed in 60 minutes $\to$ Critical Page (Immediate PagerDuty wake-up call to on-call engineer).
3x Burn Rate over 6 hours: 5% of the monthly budget is consumed over 6 hours $\to$ Warning Ticket / Slack notification during business hours.

4 Tricky Questions

1. Why is Average (Mean) latency fundamentally flawed for evaluating microservice performance?

Answer: The arithmetic mean mathematically conceals severe performance degradations suffered by users (Tail Latency):

Suppose 99 users receive responses in 10 ms, while 1 high-value user with a large shopping cart hangs for 10,000 ms: $\text{Mean Latency} = \frac{(99 \times 10) + 10\,000}{100} = 109.9\text{ ms}$
A dashboard displaying “Average Latency: 110 ms” gives engineers a false sense of security, masking the fact that 1% of users are experiencing catastrophic 10-second timeouts.
Percentiles (p95, p99, p99.9) isolate and reveal the worst-case customer experience, enabling engineers to detect thread pool queuing, garbage collection pauses, and lock contention.

2. Why is calculating the arithmetic average of p99 latency across multiple nodes mathematically invalid?

Answer: Percentiles are non-additive mathematical quantiles. You cannot calculate $\frac{\text{p99}{\text{node1}} + \text{p99}{\text{node2}}}{2}$ and obtain the cluster’s true p99:

If Node 1 processed 10,000 requests with $\text{p99} = 50\text{ ms}$, and Node 2 processed 10 requests with $\text{p99} = 5,000\text{ ms}$, taking the average of the two numbers ($(50 + 5000) / 2 = 2525\text{ ms}$) skews the metric by orders of magnitude.
The Correct Approach: In Prometheus, applications expose raw histogram bucket counts (http_server_requests_seconds_bucket). The query aggregates the raw counts across all nodes first via sum(...) by (le), and only then evaluates the quantile function across the combined distribution:
```
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le))
```

3. How do Pull-based (Prometheus) and Push-based (StatsD / Datadog) architectures compare, and when is each preferred?

Answer:

Pull Model (Prometheus): The centralized monitoring server periodically initiates HTTP GET requests to /actuator/prometheus on target instances.
- Advantages: Natural node liveness detection (if a scrape fails, the instance is down); centralized rate control (the TSDB server controls its own ingestion rate and cannot be overwhelmed by client flood).
Push Model (StatsD / OpenTelemetry Collector / Datadog): Application instances push UDP/HTTP packets out to collector daemons.
- Advantages: Essential for ephemeral, short-lived workloads like Serverless functions (AWS Lambda) or batch jobs that may execute and terminate before a Prometheus scraper polling interval (15s) ever triggers.

4. What causes Alert Fatigue, and what are the strict criteria for an Actionable Alert?

Answer: Alert Fatigue occurs when on-call engineers receive dozens of non-actionable, low-priority notifications daily. Over time, engineers become desensitized and inevitably snooze or ignore a notification during a true catastrophic outage. Criteria for an Actionable Alert:

Symptom-Based, Not Cause-Based: Alert when user-facing SLOs degrade (e.g., “Checkout error rate > 2%”), NOT when an internal metric fluctuates harmlessly (e.g., “Pod CPU hit 85%”).
Requires Urgent Human Intervention: If an alert does not require an engineer to wake up or immediately change system state, it should be a dashboard chart or daily report, not a page.
Mandatory Runbook / Playbook Link: Every alert must include a link to a documented runbook specifying initial triage commands, escalation paths, and remediation playbooks.

🎯 Interview Cheat Sheet

Must Know:

Observability Triad: Metrics (numbers / trends), Logs (discrete events / why), Traces (cross-service request execution flows).
Google SRE 4 Golden Signals: Latency, Traffic, Errors, Saturation.
Frameworks: RED (Rate, Errors, Duration) for APIs; USE (Utilization, Saturation, Errors) for infrastructure.
Metrics Cardinality Explosion: Never place high-cardinality fields (userId, orderId) in metric tags; keep tags low-cardinality.
Java Stack: Spring Boot Actuator + Micrometer scraped by Prometheus and visualized in Grafana.
Percentiles over Averages: Always use p95/p99 histograms to capture tail latencies; averages conceal severe outliers.
Reliability Framework: SLI (actual measure) $\to$ SLO (target) $\to$ Error Budget (burn rate policy).

Common Follow-up Questions:

“What is the difference between a Counter and a Gauge in Micrometer?” → A Counter is a monotonically increasing value (e.g., total requests handled); a Gauge represents an instantaneous snapshot value that can rise and fall arbitrarily (e.g., active thread count, CPU temperature, current connection pool usage).
“How does Prometheus discover new microservice pods dynamically?” → Via Kubernetes Service Discovery (kubernetes_sd_configs), which watches K8s API events for new pod and service endpoints.

Red Flags (DO NOT Say):

❌ “We evaluate microservice performance using Average Latency.” (Averages mask severe p99 tail latency outliers).
❌ “We tag our Prometheus metrics with userId and transactionId to track user behavior.” (Triggers a catastrophic cardinality explosion that crashes Prometheus with OOM).
❌ “Configure PagerDuty to wake up engineers whenever CPU utilization exceeds 80%.” (Causes severe alert fatigue; high CPU is normal if latency and error rates meet SLOs).

What is distributed tracing — Distributed trace propagation and spans
How to ensure fault tolerance of microservices — Resilience and stability
How to implement horizontal scaling of microservices — Autoscaling on Prometheus metrics
What is Circuit Breaker Pattern — Metrics-driven circuit breaking