---

🟢 Junior Level

30-Second Summary

The distinction between the three delivery semantics reflects how a distributed system responds to network failures, node crashes, and retry policies:

• At-Most-Once ($0 \le N \le 1$): Messages may be lost in transit, but are never duplicated. The producer transmits in a fire-and-forget mode without retries, and the consumer auto-commits offsets before downstream processing completes. If a crash occurs, unwritten records are lost forever.
• At-Least-Once ($N \ge 1$): Messages are guaranteed never to be lost, but may arrive duplicated. The producer retries upon failure until an acknowledgment is received, and the consumer commits offsets strictly after database persistence succeeds. This is the universal enterprise default.
• Exactly-Once ($N = 1$): Each message is processed strictly once with zero data loss and zero duplicate side-effects. Within Apache Kafka, this is coordinated atomically across topics via the Transaction API (Kafka-to-Kafka streams).

---

Real-World Analogy

• At-Most-Once — Public Address Announcement at a Train Station: The announcer reads the departure platform over the loudspeaker once. If a passing freight train drowns out the announcement, the passenger misses the information. The station never repeats it, and the passenger never hears it twice.
• At-Least-Once — Pizza Delivery Paid on Arrival: The delivery driver knocks on the door until an occupant answers. If the doorbell glitches, the driver may knock multiple times or a second delivery driver may be dispatched. The pizza is guaranteed to arrive, but the customer may deal with multiple doorbells.
• Exactly-Once — Bank Card Payment with an Idempotency Key: When you tap your card at a terminal, the POS device generates a unique cryptographic transaction UUID. Even if cellular connectivity drops mid-transmission and the terminal retries three times, the central bank ledger debits your account exactly once.

---

Failure Scenarios: Anatomy of Data Loss and Duplication

Scenario: Consumer Node Crashes Mid-Processing

At-Most-Once:
  [ Poll Record ] ──► [ Commit Offset ] ──► [ Crash 💥 ] ──► [ Save to DB ] (NEVER EXECUTED!)
  Outcome: Message is permanently lost; recovered consumer skips past the unwritten record.

At-Least-Once:
  [ Poll Record ] ──► [ Save to DB ] ──► [ Crash 💥 ] ──► [ Commit Offset ] (NEVER EXECUTED!)
  Outcome: Recovered consumer re-reads the uncommitted message and executes a second DB write (DUPLICATE).

Exactly-Once (Kafka Transaction):
  [ Poll Record + Open Tx ] ──► [ Write to Outgoing Topic ] ──► [ Atomic Offset Commit ]
  Outcome: Two-Phase Commit ensures either both operations commit or both abort (CLEAN STATE).

---

🟡 Middle Level

Comprehensive Configuration Matrix

Achieving a specific semantic guarantee requires aligning Producer, Broker, and Consumer properties:

Parameter	At-Most-Once	At-Least-Once (Default)	Exactly-Once (Kafka-to-Kafka)
Producer acks	0 (or 1)	all (wait for full ISR)	all
Producer retries	0	Integer.MAX_VALUE	Integer.MAX_VALUE
Producer Idempotence	false	true	true
Producer transactional.id	null	null	"tx-producer-01"
Broker min.insync.replicas	1	2	2
Consumer enable.auto.commit	true	false	false
Consumer isolation.level	read_uncommitted	read_uncommitted	read_committed
Commit Timing	Prior to processing	Strictly after DB persistence	Inside sendOffsetsToTransaction()

---

Production Pattern: At-Least-Once with Idempotent Consumer

Because Kafka cannot coordinate distributed Two-Phase Commit (2PC) transactions with external relational databases like PostgreSQL, enterprise systems pair At-Least-Once transport with an Idempotent Consumer:

@Service
@RequiredArgsConstructor
public class IdempotentPaymentConsumer {

    private final JdbcTemplate jdbcTemplate;

    @Transactional
    @KafkaListener(topics = "payments", groupId = "billing-group")
    public void consume(PaymentEvent event, Acknowledgment ack) {
        // Atomic Upsert utilizing PostgreSQL unique constraint on 'id'
        int rowsInserted = jdbcTemplate.update("""
            INSERT INTO payments (id, user_id, amount, status)
            VALUES (?, ?, ?, 'COMPLETED')
            ON CONFLICT (id) DO NOTHING
            """, event.id(), event.userId(), event.amount());

        if (rowsInserted == 0) {
            log.warn("Duplicate payment event detected: {}. Bypassing business execution.", event.id());
        } else {
            executeAccountBalanceCredit(event);
        }

        // Commit offset to Kafka strictly after the database transaction successfully commits
        ack.acknowledge();
    }
}

---

🔴 Senior Level

The Dual-Write Anti-Pattern and Transactional Outbox

A widespread architectural vulnerability in microservices is the naive Dual-Write:

// ❌ ANTI-PATTERN: Non-atomic Dual-Write
@Transactional
public void createOrder(Order order) {
    orderRepository.save(order); // 1. Writes to PostgreSQL
    kafkaTemplate.send("orders", order.getId(), order); // 2. Transmits to Kafka 💥
}

Why Dual-Write Violates Delivery Guarantees:

• If network transmission to Kafka times out, the Spring @Transactional annotation rolls back the database insert, but the broker may have already written the message to disk $\to$ Phantom event in Kafka with no database counterpart.
• If the database commits but the service pod crashes (OOMKilled) before kafkaTemplate.send() executes $\to$ Data saved in database but lost to the rest of the enterprise.

Architectural Solution: Transactional Outbox Pattern + Debezium CDC:

1. Within a single local ACID transaction in PostgreSQL, write the domain order to orders and the message payload to outbox_events.
2. A change data capture agent (Debezium) monitors the PostgreSQL Write-Ahead Log (WAL) and streams outbox events to Kafka with guaranteed At-Least-Once semantics.
3. Downstream consumers apply unique-key deduplication, achieving end-to-end consistency without distributed XA transactions.

---

Transaction Coordinator Internals in Exactly-Once Semantics

When a transactional producer executes in Kafka:

1. initTransactions(): Registers transactional.id with the broker’s Transaction Coordinator. The coordinator increments the producer_epoch and fences any zombie producer instances sharing that ID.
2. beginTransaction(): Opens a local client-side transaction boundary.
3. send(): Records are published with transactional metadata headers. Brokers write records to log segments, but conceal them from read_committed consumers.
4. commitTransaction():
   • Phase 1: Coordinator logs PREPARE_COMMIT to __transaction_state.
   • Phase 2: Coordinator writes COMMIT control markers across all user topic partitions and __consumer_offsets.
   • Phase 3: Coordinator logs COMPLETE_COMMIT to __transaction_state.
5. Consumers with isolation.level=read_committed observe the COMMIT marker and release buffered messages to application code.

---

4 Tricky Questions

1. Why does the naive Dual-Write pattern break delivery guarantees, and how does Transactional Outbox eliminate distributed Two-Phase Commit (XA)?

Answer:

• Dual-Write attempts to coordinate two independent persistence boundaries (PostgreSQL and Kafka) without a distributed transaction manager. If a failure occurs between the database commit and the broker transmission, state desynchronizes: records are persisted without events (lost data), or events are published while database transactions roll back (phantom data).
• The Transactional Outbox Pattern eliminates distributed coordination by executing both writes within a single local ACID transaction (inserting the domain entity and an event record into an outbox table).
• An asynchronous, battle-tested log reader (e.g., Debezium CDC reading the PostgreSQL WAL) reads the outbox table and streams records to Kafka with At-Least-Once reliability. Message loss is mathematically impossible, and downstream consumers use idempotency keys to filter potential duplicates.

---

2. Can enabling Kafka Exactly-Once Semantics (EOS) prevent duplicate processing if a mobile client taps “Submit Order” twice during a cellular network glitch?

Answer:

• No, absolutely not.
• Kafka EOS guarantees deduplication strictly within the Kafka infrastructure boundary (between the KafkaProducer client and Kafka brokers during internal network retries).
• When a user taps “Submit Order” on a phone, the mobile app sends an HTTP POST /orders. If the cellular tower drops the connection before the phone receives the 200 OK, the mobile app retries and sends a second HTTP POST /orders.
• The backend web service receives two distinct HTTP requests. It creates two distinct ProducerRecord instances with different timestamps or IDs, and Kafka processes both as independent, valid transactions.
• Preventing duplication at this boundary requires an End-to-End Idempotency Key passed from the client in the HTTP header, validated against Redis or a relational database constraint before publishing to Kafka.

---

3. What will a consumer configured with isolation.level = read_uncommitted (default) receive if a producer publishes 10 messages and subsequently calls producer.abortTransaction()?

Answer:

• The consumer will read all 10 aborted messages as if they were successfully committed!
• Kafka uses an append-only log model. Brokers never physically erase or roll back records when a transaction aborts; they simply append an ABORT control record marker at the end of the partition log.
• A consumer running with default read_uncommitted settings ignores control markers entirely and immediately returns all raw records to application code.
• To prevent consuming rolled-back transactions, consumers must explicitly configure isolation.level = read_committed.

---

4. What is the performance penalty (overhead) of upgrading from At-Least-Once to Exactly-Once Semantics in Kafka?

Answer:

• Enabling EOS incurs measurable throughput and latency trade-offs:
  1. Latency Inflation (2x–3x): Consumers with read_committed cannot read records at the physical head of the partition log. They must wait for the producer to complete its transaction, for the coordinator to append a COMMIT marker, and for the Last Stable Offset (LSO) to advance.
  2. Write Amplification: Every committed transaction writes two additional control markers per topic partition and state transitions to the __transaction_state topic with forced disk flushes (fsync).
  3. Throughput Degradation: Micro-transactions (committing on every individual message) can collapse producer throughput by up to 70–80%. High-throughput EOS requires batching records over a temporal window (e.g., transactions every 100–500 ms).

---

🎯 Interview Cheat Sheet

Core Concepts at a Glance

• At-Most-Once: Fastest, loss tolerated, no duplicates ($0 \le N \le 1$).
• At-Least-Once: Enterprise standard, zero loss, duplicates possible ($N \ge 1$).
• Exactly-Once: Zero loss, zero duplicates ($N = 1$), strictly within Kafka-to-Kafka boundaries.
• External Storage Myth: Kafka Transactions do not extend into PostgreSQL or external APIs.
• Dual-Write Danger: Never write sequentially to DB and Kafka in a single application method; use Transactional Outbox.
• Consumer Isolation: Default read_uncommitted consumes aborted transactions; EOS requires read_committed.
• Durability Baseline: acks=all with min.insync.replicas >= 2 is mandatory for both At-Least-Once and EOS.

---

Delivery Guarantee Comparison Matrix

Attribute	At-Most-Once	At-Least-Once	Exactly-Once (Kafka EOS)
Data Loss	Possible	❌ Zero	❌ Zero
Duplicate Delivery	❌ Zero	Possible	❌ Zero
End-to-End Latency	Baseline ($1\times$)	Low ($1.1 - 1.2\times$)	High ($2 - 3\times$)
System Throughput	$100\%$	$85 - 95\%$	$50 - 75\%$
Typical Domain	Metrics, telemetry, logs	E-commerce, payment events	Financial ledgers, stream processing

---

Red Flags to Avoid

• ❌ “Kafka Exactly-Once ensures our database will never receive duplicate orders from mobile clients.” (Client retries require API-level idempotency keys).
• ❌ “Setting enable.auto.commit=true guarantees At-Least-Once processing.” (Auto-commit runs on a timer and commits offsets before processing completes, causing message loss).
• ❌ “Configuring transactions on the producer automatically makes the consumer Exactly-Once.” (Consumers must explicitly set isolation.level=read_committed).
• ❌ “Distributed 2PC / XA between Kafka and PostgreSQL is the recommended microservice design.” (Anti-pattern; introduces blocking deadlocks and degrades performance).

---

Related Topics

• What Message Delivery Guarantees Does Kafka Provide — Delivery guarantee fundamentals
• How to Configure Exactly-Once Semantics — Step-by-step EOS configuration
• What is Idempotent Producer — Sequence numbers and deduplication
• What is Producer Acknowledgment and What Modes Exist (acks=0, 1, all) — Producer ACKs mechanics