📨 Section 15 · Question #9

What Message Delivery Guarantees Does Kafka Provide

Apache Kafka supports three classic message delivery semantics:

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

Apache Kafka supports three classic message delivery semantics:

  1. At-Most-Once:
    • Messages may be lost during network or broker failures, but are never duplicated.
    • Use Case: High-frequency metrics, sensor telemetry, and clickstream logging where sporadic data loss is preferable to duplicate delivery or retry latency.
  2. At-Least-Once (Enterprise Standard):
    • Messages are guaranteed never to be lost, but may arrive duplicated during client retries or network partition reconnects.
    • Use Case: The default industrial standard across 95% of enterprise microservices, paired with idempotent consumer processing.
  3. Exactly-Once (EOS):
    • Messages are delivered and processed strictly once with zero data loss and zero duplicates.
    • Architectural Limitation: Supported out of the box strictly within the Kafka boundary (Kafka-to-Kafka streaming: consume from Topic A $\to$ process in memory $\to$ write to Topic B and commit offsets within an atomic transaction).

Real-World Analogy

  • At-Most-Once (Regular Untracked Mail): Dropping an envelope into a street postal box. If rain destroys the letter, it is lost forever. You never receive confirmation, but the recipient will never receive duplicate letters.
  • At-Least-Once (Courier Delivery with Signature): A courier delivers a parcel to your doorstep. You accept the delivery, but the courier’s electronic scanner freezes during confirmation. The courier leaves, and the warehouse sends a second courier with a duplicate parcel the following morning. The delivery is guaranteed, but a duplicate occurred.
  • Exactly-Once (Interbank Wire Clearing): Transferring $1,000 between two accounts within a closed central banking ledger. Funds are debited from Account A and credited to Account B in a single atomic transaction; the transfer cannot leave funds duplicated or destroyed.

Configuration Overview Table

Semantic Tier Producer Configuration Consumer Configuration Primary Risk
At-Most-Once acks=0 or acks=1, retries=0 enable.auto.commit=true (commits before processing) ❌ Message Loss
At-Least-Once acks=all, retries=MAX, enable.idempotence=true enable.auto.commit=false (manual commit after processing) ⚠️ Duplicate Delivery
Exactly-Once transactional.id=TX-1, enable.idempotence=true isolation.level=read_committed, manual commit ⏳ Latency Overhead (~10-20%)

🟡 Middle Level

The End-to-End Delivery Contract

A delivery guarantee is an end-to-end contract spanning the producer, the broker cluster, and the consumer. Misconfiguring any single link collapses the entire guarantee:

\[\text{End-to-End Guarantee} = \text{Producer Guarantee} + \text{Broker Durability} + \text{Consumer Processing}\] 
[ Producer ]                    [ Kafka Broker ]                    [ Consumer ]
 acks=all                        min.insync.replicas=2               enable.auto.commit=false
 enable.idempotence=true         replication.factor=3                commitSync() AFTER database commit
  1. Producer: Verifies messages are written to the broker quorum. Retries automatically upon network faults.
  2. Broker: Enforces data durability on disk and synchronizes writes across In-Sync Replicas (min.insync.replicas).
  3. Consumer: Commits offsets to __consumer_offsets strictly after downstream business logic and persistence succeed.

Spring Boot 3 / Java 21 At-Least-Once Implementation

@Configuration
public class KafkaProducerConfig {

    @Bean
    public ProducerFactory<String, Object> producerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.ACKS_CONFIG, "all"); // Wait for full ISR acknowledgment
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // Eliminate retry duplicates
        props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
        return new DefaultKafkaProducerFactory<>(props);
    }
}

@Service
public class OrderProcessingConsumer {

    @KafkaListener(topics = "orders", groupId = "order-group")
    public void consume(ConsumerRecord<String, String> record, Acknowledgment ack) {
        // 1. Execute and commit business database transaction first
        orderRepository.save(parseRecord(record.value()));
        
        // 2. Acknowledge offset to Kafka strictly after database persistence succeeds
        ack.acknowledge();
    }
}

🔴 Senior Level

Anatomy of Kafka Transactions (KIP-98 Exactly-Once)

Achieving Exactly-Once Semantics (EOS) within Kafka relies on three architectural components:

  1. Idempotent Producer:
    • The broker assigns each producer instance a 64-bit Producer ID (PID).
    • Each message receives a monotonically incrementing Sequence Number.
    • The broker caches the latest 5 sequence numbers per PID in memory. If a retry sends an already-persisted sequence number, the broker returns an ACK without duplicating the log entry.
  2. Transaction Coordinator & __transaction_state Topic:
    • A dedicated broker coordinates the Two-Phase Commit (2PC) protocol across participating topic partitions and records state transitions to the compacted internal topic __transaction_state.
  3. Control Records (Transaction Markers):
    • When a transaction commits or aborts, the coordinator writes a non-data control record (COMMIT or ABORT) into each partition log.

The Dual-Write Reality: Why Kafka Cannot Guarantee EOS into PostgreSQL

A common interview pitfall is claiming: “We enabled Kafka Transactions, so our records are written to PostgreSQL exactly once.”

  • This is false. Kafka brokers and external relational databases (PostgreSQL, MySQL, Oracle) do not participate in a shared distributed XA Two-Phase Commit transaction.
  • If a consumer inserts a row into PostgreSQL and crashes before committing its offset to Kafka, the recovered consumer will re-read the message upon restart and execute a duplicate INSERT.
  • Enterprise Solutions for End-to-End EOS:
    1. Idempotent Consumer: Use database unique constraints and idempotent writes (INSERT ... ON CONFLICT (payment_id) DO NOTHING).
    2. Transactional Inbox Pattern: Save the incoming event ID into a local database table (inbox_messages) within the same local ACID transaction as the business entity. Duplicate events violate the primary key constraint and are discarded.

Stalled Consumers: Last Stable Offset (LSO)

Consumers configured with isolation.level=read_committed read partition logs only up to the Last Stable Offset (LSO):

  • LSO is defined as the offset of the earliest uncommitted (in-flight) transaction in that partition.
  • If a transactional producer opens a transaction at offset 1,000 and hangs for 10 minutes, all subsequent committed records from other producers (offsets 1,001+) are blocked from consumption.
  • Consumers cannot advance past offset 1,000 until the hanging transaction times out (transaction.timeout.ms) and the coordinator appends an ABORT marker.

4 Tricky Questions

1. Why can’t Kafka guarantee Exactly-Once out of the box when reading from a topic and persisting into PostgreSQL, and how must an engineer resolve this?

Answer:

  • Kafka’s Transaction Coordinator controls commits only across Kafka topic partitions and the __consumer_offsets topic. It cannot coordinate atomic Two-Phase Commit commits with external database engines (PostgreSQL, Redis, MongoDB).
  • If the consumer commits the SQL transaction and then loses network connectivity or suffers an OOM crash before committing its offset to Kafka, the newly assigned consumer will re-fetch the uncommitted record and duplicate the SQL operation.
  • Production Resolution:
    1. Design idempotent consumers using SQL Upsert semantics (INSERT ... ON CONFLICT (order_id) DO NOTHING).
    2. Implement the Transactional Inbox Pattern: insert the message payload and an event log entry with a unique constraint (message_id PRIMARY KEY) within a single local database transaction. Subsequent duplicate deliveries trigger a constraint violation and are safely ignored.

2. What is Last Stable Offset (LSO), and how can a stalled producer cause a Denial of Service for consumers using isolation.level=read_committed?

Answer:

  • LSO (Last Stable Offset) is the offset of the earliest unresolved transaction in a partition.
  • Consumers with isolation.level=read_committed are strictly forbidden from reading past the LSO to ensure they never consume aborted transaction data that might later roll back.
  • If Producer A begins a transaction at offset 500 and hangs (e.g., deadlock, slow GC, or network partition), the LSO freezes at 500. Even if Producers B, C, and D write and commit millions of messages at offsets 501–1,000,000, read_committed consumers cannot read a single message past 500.
  • Consumers appear completely frozen, and consumer lag explodes until the broker forcibly aborts Producer A’s transaction after transaction.timeout.ms (default 60 seconds), writing an ABORT marker that advances the LSO.

3. Why are enable.idempotence=true and acks=all enabled by default in Kafka 3.0+, and is it still possible to lose messages when acks=all is configured?

Answer:

  • Kafka 3.0 adopted “Secure and Resilient Defaults” because hardware and network reliability have improved to the point where idempotent retries add negligible throughput overhead compared to the cost of data corruption.
  • How to lose data with acks=all:
    • If the broker configuration min.insync.replicas is set to 1 (the default in older installations).
    • acks=all requires confirmation from all replicas currently present in the In-Sync Replicas (ISR) list. If broker network hiccups cause followers to drop out of ISR, the ISR shrinks to only the Leader.
    • The leader broker writes to its local disk and sends an immediate ACK to the producer. If that leader node suffers catastrophic hardware or power failure before followers re-sync, the acknowledged record is lost permanently!
    • Rule for True Durability: Always set replication.factor >= 3 and min.insync.replicas >= 2.

4. What is the fundamental difference between commitSync() and commitAsync() in At-Least-Once consumers, and why are they typically combined?

Answer:

  • commitSync() blocks the consumer thread until the broker responds with an acknowledgment. It automatically retries on retriable errors. It guarantees that offsets are committed before proceeding, but introduces network latency pauses that reduce throughput.
  • commitAsync() transmits the offset commit request non-blockingly and does not retry on failure. If a commit for offset 200 fails due to a network glitch, and a subsequent commit for offset 300 succeeds, retrying offset 200 would cause an out-of-order regression that rolls back consumer progress.
  • Production Best Practice: In the main polling loop, invoke fast non-blocking commits (commitAsync()), and in the finally block or ConsumerRebalanceListener.onPartitionsRevoked(), execute a blocking commitSync() to guarantee that final offsets are durably persisted before shutdown or partition migration.

🎯 Interview Cheat Sheet

Core Concepts at a Glance

  • At-Most-Once: Messages may be dropped, never duplicated (acks=0, auto-commit before processing).
  • At-Least-Once (Default): Zero message loss, potential duplicates (acks=all, min.insync.replicas=2, manual commit after processing).
  • Exactly-Once (EOS): Restricted to Kafka-to-Kafka topologies via Transaction API (transactional.id, isolation.level=read_committed).
  • End-to-End Integrity: True guarantees require coordinated alignment across Producer, Broker, and Consumer.
  • External Storage Integration: Kafka cannot 2PC with databases; use Idempotent Writes or Transactional Inbox.
  • LSO Blocking: Incomplete transactions freeze consumption for all read_committed consumers up to transaction.timeout.ms.
  • Durability Invariant: acks=all guarantees durability only when min.insync.replicas >= 2.

Delivery Semantic Trade-Off Matrix

Semantic Level Data Loss Risk Duplicate Risk Throughput Complexity
At-Most-Once High None Maximum Minimal
At-Least-Once None Possible High Low (requires consumer idempotency)
Exactly-Once (EOS) None None Moderate (-10-20%) High (2PC transaction coordination)

Red Flags to Avoid

  • ❌ “Kafka Transactions ensure exactly-once writes to PostgreSQL automatically.” (Kafka does not manage database transactions).
  • ❌ “Setting acks=all eliminates message loss in all cluster configurations.” (If min.insync.replicas=1, single-replica failure causes data loss).
  • ❌ “Configuring enable.auto.commit=true provides At-Least-Once delivery.” (Auto-commit runs on a timer and commits unread or uncompleted offsets, causing data loss).
  • ❌ “Kafka Exactly-Once eliminates the need for database unique constraints.” (External systems must remain idempotent).