How to organize communication between microservices
In a microservices architecture, services run in isolated processes and communicate over the network. The choice of communication mechanism defines the overall system's resilien...
🟢 Junior Level
In a microservices architecture, services run in isolated processes and communicate over the network. The choice of communication mechanism defines the overall system’s resilience, latency, and scalability.
Synchronous (Request / Response):
[Client Service] ──HTTP / gRPC──► [Target Service]
◄──JSON / Proto──┘ (Caller blocks/waits for immediate response)
Asynchronous (Event-Driven / Pub-Sub):
[Producer Service] ──Publish──► [Message Broker (Kafka / RabbitMQ)]
│
├──Consume──► [Consumer Service A]
└──Consume──► [Consumer Service B]
(Producer does not wait; decoupled in time)
The Two Fundamental Communication Styles
- Synchronous Communication (Point-to-Point): The caller sends an HTTP/REST or gRPC request and halts execution while awaiting an immediate response. This introduces Temporal Coupling: both the caller and recipient must be simultaneously online and responsive. If the target service fails or suffers network latency, the calling request fails or times out.
- Asynchronous Communication (Event-Driven): The sender publishes a message or business domain event to an intermediate message broker (Apache Kafka, RabbitMQ). Consumers process messages independently at their own pace. The sender does not know or care when the event will be processed, achieving temporal and architectural decoupling.
🟡 Middle Level
1. Technology Comparison Matrix
| Technology | Style & Topology | Data Payload | Production Sweet Spot |
|---|---|---|---|
| REST over HTTP/1.1 | Synchronous (1-to-1) | JSON / XML | Public edge APIs for mobile/web browsers; simple CRUD integrations. |
| gRPC over HTTP/2 | Synchronous & Streaming | Protocol Buffers (Binary) | ⚡ Internal Service-to-Service RPC: Ultra-low latency, high throughput, and polyglot type safety. |
| Apache Kafka | Asynchronous (Pub/Sub) | Avro / Protobuf / JSON | High-throughput event-carried state transfer, audit logging, Event Sourcing, and Change Data Capture (CDC). |
| RabbitMQ | Asynchronous (Broker-Routed) | Any (Binary / JSON) | Complex AMQP message routing, task worker queues, and delayed Dead-Letter Exchange (DLX) processing. |
2. Spring Boot 3 Implementation Patterns
A. Modern Synchronous REST with RestClient (Spring 6+)
@Service
@RequiredArgsConstructor
public class UserClient {
private final RestClient restClient;
public UserProfileDTO getUserProfile(Long userId) {
return restClient.get()
.uri("http://user-service/api/v1/users/{id}", userId)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.onStatus(HttpStatusCode::is4xxClientError, (req, resp) -> {
throw new UserNotFoundException("User not found: " + userId);
})
.body(UserProfileDTO.class);
}
}
B. High-Speed Binary gRPC Contract (.proto)
syntax = "proto3";
option java_multiple_files = true;
package com.example.grpc;
service AccountService {
rpc GetBalance (BalanceRequest) returns (BalanceResponse);
}
message BalanceRequest {
string account_id = 1;
}
message BalanceResponse {
int64 amount_cents = 1;
string currency = 2;
}
C. Asynchronous Event-Driven Messaging with Kafka
@Service
@RequiredArgsConstructor
public class OrderEventProducer {
private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;
public void publishOrderCreated(Order order) {
OrderCreatedEvent event = new OrderCreatedEvent(order.getId(), order.getTotalCost());
// Partition key guarantees strict message ordering per orderId:
kafkaTemplate.send("order-events", order.getId().toString(), event);
}
}
🔴 Senior Level
1. The SLA Availability Multiplication Law
Chaining synchronous microservice calls creates an exponential degradation in system reliability known as SLA Multiplication:
[Gateway] ──► [Service A (99.5%)] ──► [Service B (99.5%)] ──► [Service C (99.5%)] ──► [Service D (99.5%)]
If a user request depends synchronously on a chain of 4 microservices, each boasting an individual availability SLA of 99.5%: \(\text{SLA}_{\text{system}} = 0.995 \times 0.995 \times 0.995 \times 0.995 \approx 98.0\%\)
- At $99.5\%$ availability, an individual service experiences $1.8\text{ days}$ of downtime per year.
- At $98.0\%$ composite availability, the chained system suffers over $7.3\text{ days}$ of total downtime per year!
[!IMPORTANT] Architectural Guardrail: Never allow synchronous call chains to exceed a depth of 2. Deeper dependencies must be decoupled using asynchronous events, data denormalization, or CQRS materialized views.
2. Why gRPC Outperforms REST/JSON by 5x–10x
REST (HTTP/1.1 + JSON):
[TCP Handshake] ──► [Verbose Text Headers (1 KB)] ──► [Text JSON: {"id":123}]
(Repeated TCP handshakes, string allocations, and heavy text serialization on JVM heap)
gRPC (HTTP/2 + Protocol Buffers):
[Single Persistent TCP Connection] ──► [HPACK Binary Headers] ──► [Varint Binary: 0x08 0x7B]
(Stream multiplexing, zero string allocations, compact native byte decoding)
- Varint Binary Serialization: Numeric integers are encoded in compact variable-length bytes (a 32-bit integer occupies 1–2 bytes instead of 10 ASCII characters like
"2147483647"). - HTTP/2 Connection Multiplexing: Hundreds of concurrent RPC requests travel over a single shared TCP connection, eliminating the CPU and latency overhead of repeated TCP handshakes and TLS renegotiation.
- Rigid Schema Evolution: Numerical field tags (
tag numbers) in.protofiles ensure safe backward and forward compatibility without parsing schema metadata on every request.
3. Asynchronous Request-Reply Pattern
When an operation requires temporal decoupling but the caller ultimately needs a correlated response:
[Service A] ──1. Send Command (CorrelationId=42, ReplyTo="order-replies")──► [Kafka: commands]
│ │
│ 3. Match Future by CorrelationId ▼
◄──4. Complete Response (CorrelationId=42)──────── [Kafka: order-replies] ◄── [Service B]
In Spring Kafka, this is implemented using ReplyingKafkaTemplate: the originating thread registers a CompletableFuture indexed by CorrelationId in an in-memory concurrent map and suspends, awaiting a matched response from a dedicated reply topic with a hard timeout.
4 Tricky Questions
1. How do you propagate Distributed Tracing context across heterogeneous protocols (HTTP, gRPC, and Kafka)?
Answer: Distributed tracing across diverse network protocols relies on the W3C TraceContext standard:
- HTTP/REST: Propagated via the standard HTTP header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01(containing version, traceId, parentSpanId, and traceFlags). - gRPC: Injected into gRPC
Metadata(binary/ASCII key-value pairs transmitted in HTTP/2 headers) via an automatedClientInterceptor. - Kafka: Injected into binary record headers:
record.headers().add("traceparent", bytes). Tracing frameworks (OpenTelemetry and Micrometer Tracing) automate this context injection and extraction transparently across client wrappers and message listeners.
2. What is the “Tolerant Reader” pattern and how does it prevent contract breakage during API evolution?
Answer: The Tolerant Reader pattern dictates that a consuming service must be strict in what it sends, but extraordinarily lenient and liberal in what it receives:
- Ignore Unknown Properties: In Jackson, configure
@JsonIgnoreProperties(ignoreUnknown = true)so new fields added by a producer do not cause deserialization failures. - Avoid Positional Assumptions: Never assume a specific ordering of fields in JSON arrays or objects.
- Protobuf Tag Reservations: In Protobuf, never modify or reuse existing numerical tag numbers. When deprecating a field, declare it as
reserved(e.g.,reserved 4, 12;) to prevent future developers from assigning those tag numbers to new types.
3. In which concrete scenarios is RabbitMQ superior to Apache Kafka for inter-service communication?
Answer:
- Complex Routing Topologies: RabbitMQ supports dynamic routing at the broker layer via Topic, Fanout, Direct, and Headers exchanges. Kafka requires consumers to ingest all partition data and execute filtering in application code.
- Granular Per-Message Acknowledgments: RabbitMQ allows individual messages to be selectively acknowledged, rejected, or re-queued (
basic.reject(requeue=true)), or routed to a Dead-Letter Exchange (DLX) based on custom TTLs. In Kafka, consumers commit offsets sequentially; you cannot reject message #42 without affecting messages #43 and #44. - Dynamic Task Queues (Work Pools): Distributing tasks among 100 worker instances is seamless in RabbitMQ. In Kafka, the maximum number of parallel active consumer pods in a consumer group is strictly capped by the number of partitions in the topic.
4. What is TCP-level Head-of-Line (HoL) Blocking in HTTP/2 and gRPC, and how is it resolved in HTTP/3?
Answer: In HTTP/2 and gRPC, multiple logical request streams are multiplexed across a single physical TCP connection. At the application layer, streams are independent; however, at the OS network transport layer, TCP enforces strict byte-stream ordering. If a single IP packet containing bytes from Stream A is dropped in transit, the operating system kernel freezes delivery of all other streams (Streams B, C, and D) on that connection until the missing TCP segment is retransmitted and acknowledged. On unreliable or congested networks (mobile cellular, lossy Wi-Fi), this can make HTTP/2 slower than HTTP/1.1. HTTP/3 solves this by replacing TCP with QUIC over UDP. In QUIC, stream multiplexing occurs natively at the transport layer, ensuring packet loss on one stream has zero impact on any other concurrent stream.
🎯 Interview Cheat Sheet
Must Know
- Two Core Styles: Synchronous (REST, gRPC) for immediate queries; Asynchronous (Kafka, RabbitMQ) for decoupled state mutations and events.
- gRPC Efficiency: 5x–10x faster than REST/JSON due to Protobuf binary serialization, HTTP/2 multiplexing, and zero JSON text-parsing overhead.
- Kafka vs RabbitMQ: Kafka is an immutable distributed commit log (high-throughput event streaming, replayability); RabbitMQ is a message broker (task queues, complex routing, per-message ACKs).
- The Chaining Hazard: Chaining synchronous calls multiplies failure rates ($\prod \text{SLA}_i$); keep synchronous depth $\le 2$.
- Idempotency Mandate: Because message brokers guarantee at-least-once delivery, every asynchronous message consumer must be strictly idempotent.
Protocol Decision Matrix
| Requirement | Recommended Protocol | Reason |
|---|---|---|
| User Login / Authentication | gRPC or REST | Requires immediate synchronous confirmation. |
| High-Throughput Internal RPC | gRPC | Binary protobuf speed and multiplexing. |
| Order Placed Event Propagation | Apache Kafka | Multiple consumers; requires event replay & audit log. |
| Background Email/PDF Generation | RabbitMQ / AWS SQS | Dedicated task queue with DLQ and per-task retries. |
Common Follow-up Questions
- “What is Event-Carried State Transfer?” $\to$ Including all essential entity data in the event payload so downstream consumers do not need to query the producer back over REST.
- “How do you trace calls across Kafka?” $\to$ OpenTelemetry injects the W3C
traceparentcontext into Kafka record headers. - “Why not use gRPC for public web browsers?” $\to$ Standard web browsers do not expose fine-grained HTTP/2 framing controls; requires gRPC-Web proxies.
Red Flags (DO NOT Say)
- ❌ “Synchronous REST is always preferred because managing Kafka is too complicated.” (Creates tight temporal coupling and causes cascading cluster-wide failures).
- ❌ “Execute multiple sequential REST calls in a single HTTP endpoint handler.” (Compounds latency; parallelize independent calls using Java 21 Virtual Threads).
- ❌ “Kafka guarantees exactly-once processing out of the box without consumer deduplication.” (Kafka provides at-least-once delivery; consumers must enforce idempotency via deduplication tables).
Related Topics
- What is the Difference Between Synchronous and Asynchronous Communication — Protocol deep dive
- What is Circuit Breaker Pattern — Protecting synchronous dependencies
- What is Database per Service Pattern — Service data isolation
- How to Ensure Fault Tolerance of Microservices — Resilience engineering