What is distributed tracing

In a distributed topology, a single user click can trigger 10 to 20 downstream network calls across different hosts.

🟢 Junior Level

Distributed Tracing is an observability technique used to profile, visualize, and monitor the end-to-end execution journey of a single user request as it traverses a complex distributed graph of microservices, databases, caches, and message brokers.

Client: POST /checkout
       │
       ▼
[API Gateway] ─────────────── (Total End-to-End Latency: 240 ms) ─────────┐
  │                                                                       │
  ├──► [Order Service] ────── (180 ms) ───────────────────────────────┐   │
  │      │                                                            │   │
  │      ├──► [Payment Service] ─── (120 ms: BOTTLENECK IDENTIFIED! 🐢) ──┤   │
  │      │                                                            │   │
  │      └──► [Inventory Service] ─ (30 ms) ──────────────────────────┘   │
  │                                                                       │
  └──► [Notification Service] (Asynchronous via Kafka: 15 ms) ────────────┘

Why Traditional Logging Fails in Microservices

In a distributed topology, a single user click can trigger 10 to 20 downstream network calls across different hosts.

Isolated Log Files: If a request slows down or fails, independent log streams scattered across dozens of container instances are impossible to correlate manually.
Waterfall Latency Visibility: Distributed tracing provides a single visual timeline (Waterfall Chart) showing exactly which microservice, database query, or external third-party API introduced latency, down to the exact millisecond and line of code.

🟡 Middle Level

1. Tracing Anatomy: Trace vs Span (Google Dapper Model)

The data model for distributed tracing originates from Google’s seminal Dapper (2010) research paper:

Trace (Call Tree, Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736)
├── Span A: HTTP POST /checkout (Root Span, API Gateway)
│   ├── Span B: OrderService.createOrder()
│   │   ├── Span C: SQL INSERT INTO orders
│   │   └── Span D: HTTP POST /charge (PaymentService)
│   │       └── Span E: External Stripe API Call
│   └── Span F: Kafka.send("order-events")

Trace: The complete Directed Acyclic Graph (DAG) of spans representing an entire end-to-end distributed transaction from initial client ingress to final response. Identified by a globally unique 128-bit Trace ID.
Span: A single contiguous, timed unit of work executed within an individual service boundary. A span contains:
- Operation Name: E.g., SELECT * FROM orders or POST /charge.
- Timestamps: Start time and end time (calculating precise Duration).
- Span ID & Parent Span ID: Hex identifiers establishing hierarchical parent-child relationships.
- Attributes (Tags): Key-value metadata describing the span (e.g., http.status_code=200, db.system=postgresql).
- Events (Logs): Timestamped textual annotations recording discrete milestones within the span.
- Status: OK or ERROR (with captured exception class and stack trace).

2. Context Propagation Protocol: W3C TraceContext

To maintain trace continuity across heterogeneous microservices, tracing context is propagated across network boundaries using the standard W3C TraceContext HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │                  │                      │          │
           Version            Trace ID              Parent Span ID  Flags (01 = Sampled)

3. Spring Boot 3 Implementation: Micrometer Tracing & OpenTelemetry

[!WARNING] Spring Cloud Sleuth is deprecated and removed in Spring Boot 3.x! The modern enterprise standard is Micrometer Tracing with an OpenTelemetry (OTel) or Brave bridge:

<!-- pom.xml: Spring Boot 3 Micrometer Tracing with OpenTelemetry -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Programmatic Custom Span Instrumentation:

@Service
@RequiredArgsConstructor
public class PaymentProcessingService {

    private final Tracer tracer; // io.micrometer.tracing.Tracer

    public PaymentResult processPayment(PaymentOrder order) {
        // Create an explicit child span to isolate business execution latency
        Span newSpan = tracer.nextSpan().name("fraud-check-and-charge").start();
        try (Tracer.SpanInScope ws = tracer.withSpan(newSpan.start())) {
            newSpan.tag("order.id", order.id().toString());
            newSpan.tag("order.amount", order.amount().toPlainString());

            return executeBankingTransaction(order);
        } catch (Exception ex) {
            newSpan.error(ex); // Automatically records exception metadata in the span
            throw ex;
        } finally {
            newSpan.end(); // Stops the clock and flushes telemetry
        }
    }
}

🔴 Senior Level

1. Sampling Strategies at Scale: Head-Based vs Tail-Based Sampling

At 100,000 requests per second, storing 100% of distributed traces on disk causes massive storage bloat and saturates internal network bandwidth:

Head-Based Sampling (Decided at Ingress):
[Client] ──► [API Gateway] ──(Random 1% Coin Flip)──► Traced
(Major Flaw: A rare 1-in-a-million HTTP 500 error has a 99% probability of being discarded!)

Tail-Based Sampling (Decided at Egress via OpenTelemetry Collector):
[Microservices] ──(Stream 100% of spans to local memory buffer)──► [OpenTelemetry Collector]
                                                                        │
                                                                        ├── Contains Error? ────► PERSIST TO DB! ✅
                                                                        ├── Duration > 1.5s? ───► PERSIST TO DB! ✅
                                                                        └── Fast 200 OK Trace ──► Discard 99%

Strategy	Decision Point	Advantages	Disadvantages
Head-Based Sampling	Decided at the root ingress point (API Gateway).	Simple, low CPU/memory overhead on services.	Discards critical rare errors and anomalous latency spikes before outcome is known.
Tail-Based Sampling	Decided at the OpenTelemetry Collector after the trace completes.	Guarantees 100% retention of errors and p99 anomalies while discarding uninteresting successful requests.	Requires running collector infrastructure with in-memory buffering.

2. Context Propagation Pitfalls: MDC and Reactive Threads

Tracing frameworks inject traceId and spanId into SLF4J’s MDC (Mapped Diagnostic Context) so that structured log entries include correlation IDs ([traceId=4bf92f..., spanId=00f067...]). Because MDC relies under the hood on ThreadLocal, trace context is silently lost when switching threads:

When submitting tasks to CompletableFuture.supplyAsync() or a custom ExecutorService.
Inside Spring @Async methods.
In reactive architectures (Spring WebFlux / Project Reactor), where request processing hops across non-blocking Netty Event Loop threads.

The Modern Spring Boot 3 Solution:

Use Micrometer Context Propagation:

@Configuration
public class AsyncThreadPoolConfig {

    @Bean
    public TaskExecutor taskExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(10);
        // Automatically copies MDC and trace context across thread boundaries
        executor.setTaskDecorator(new ContextPropagatingTaskDecorator());
        executor.initialize();
        return executor;
    }
}

For Project Reactor applications, enable automatic context propagation globally:

Hooks.enableAutomaticContextPropagation();

4 Tricky Questions

1. How is the W3C TraceContext `traceparent` header structured, and what do each of its four delimited fields represent?

Answer: The traceparent header is formatted as four hyphen-delimited fields: version-trace_id-parent_id-trace_flags:

version (2 hex characters): Current standard version is 00. (ff is reserved for future revisions).
trace_id (32 hex characters / 16 bytes): Globally unique identifier for the entire distributed transaction. Cannot be all zeros (00000000000000000000000000000000).
parent_id (16 hex characters / 8 bytes): Identifier of the caller’s active span that spawned the outbound request.
trace_flags (2 hex characters / 8-bit bitmap): Controls tracing behavior. The least significant bit 01 signifies Sampled (the upstream service has chosen to record this trace, instructing downstream services to record it as well).

2. Why is Tail-Based Sampling in an OpenTelemetry Collector fundamentally superior to Head-Based sampling inside application code?

Answer: When an application makes a sampling decision at the start of a request (Head-Based), it has zero foresight regarding whether that specific transaction will succeed, fail with an unexpected null pointer exception, or hang for 10 seconds on a database lock. If the service samples 5% of traffic, it has a 95% chance of silently discarding the exact trace needed to debug an outage. Tail-Based Sampling defers the decision until the entire trace graph finishes executing:

Microservices stream 100% of spans via lightweight OTLP (gRPC) to a local OpenTelemetry Collector daemon.
The collector holds spans in an in-memory ring buffer until the root span terminates.
The collector applies intelligent filtering rules: if any span in the trace has status.code = ERROR or duration > SLA_THRESHOLD, the entire trace is preserved and written to persistent storage (Jaeger, Grafana Tempo). Normal routine traces are aggressively sampled at 1%, maximizing diagnostic value while minimizing storage costs.

3. What are the concrete performance overheads of distributed tracing in Highload systems, and how are they minimized?

Answer:

CPU Utilization: Generating 128-bit cryptographically random IDs, retrieving high-resolution system clock timestamps via System.currentTimeMillis() / clock_gettime, and serializing span data structures.
JVM Heap Allocation: Instantiating Span, Scope, and attribute map objects generates garbage collection pressure on the Eden space.
Network Overhead: In high-volume microservices, transmitting detailed span metadata over HTTP/JSON can consume more bandwidth than the application payload itself. Mitigation: Use lightweight binary Protobuf over gRPC (OTLP), buffer spans in memory using a non-blocking BatchSpanProcessor executed on a background daemon thread, and apply intelligent tail-based sampling.

4. What is the technical difference between a Span Tag (Attribute) and a Span Log (Event)?