📨 Section 15 · Question #20

What is Producer Acknowledgment and What Modes Exist (acks=0, 1, all)

Starting with Kafka 3.0, the default configuration changed from acks=1 to acks=all combined with producer idempotence enabled by default (enable.idempotence=true).

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

Producer Acknowledgment (acks) is a fundamental producer configuration parameter in Apache Kafka that determines how many partition replicas must persist a message to their local log before the broker returns a success confirmation to the producer client:

  1. acks=0 (Fire-and-Forget): The producer does not wait for any response from the broker. The record is treated as sent the moment it is written to the local OS network socket buffer. Provides maximum throughput and lowest latency at the cost of high data loss risk.
  2. acks=1 (Leader Only): The producer waits for confirmation strictly from the partition Leader broker. If the Leader appends the record to its local log and crashes before follower replicas replicate it, the record is permanently lost.
  3. acks=all (or acks=-1) (Leader + ISR Quorum): The producer blocks until the Leader and all currently active In-Sync Replicas (ISR) have written the record to their logs. Guarantees zero data loss when paired with min.insync.replicas >= 2.

Starting with Kafka 3.0, the default configuration changed from acks=1 to acks=all combined with producer idempotence enabled by default (enable.idempotence=true).

Real-World Analogy

  • acks=0 (Dropping a letter into a street mailbox): You drop the letter into the slot and walk away. There is no postal receipt; if the mailbox is vandalized or burns down, the letter is lost forever.
  • acks=1 (Handing a package directly to the courier): The courier signs your receipt. Fast and personal, but if the courier’s delivery van gets into an accident before reaching the distribution sorting facility, your parcel is lost.
  • acks=all (Bank deposit box requiring three separate keys): The bank vault manager (Leader) and all authorized co-signers (ISR) verify and stamp the deposit in their independent physical logbooks. Only after every co-signer confirms the entry is a certificate of deposit issued to the customer.

Java Producer Configuration

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// Enterprise Production Durability Standard
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

🟡 Middle Level

Architectural Mechanics of ACKS Modes

acks=0:
  Producer ---- [Message] ----> (Socket Buffer) ----> Immediate Success! (No broker response)

acks=1:
  Producer ---- [Message] ----> Broker Leader (Append to PageCache) ----> Success!
                                      |
                               (Followers replicate asynchronously in background)

acks=all:
  Producer ---- [Message] ----> Broker Leader (Log.appendAsLeader)
                                      |-- Replicated to Follower 1 (ACK)
                                      |-- Replicated to Follower 2 (ACK)
                                      v
                               Only after all ISR members confirm ----> Success!

Why acks=all Fails Without min.insync.replicas

The acks=all parameter guarantees acknowledgment from all members of the active ISR set, not the total configured replication.factor.

  • Suppose a topic has replication.factor = 3, but two follower brokers crash.
  • The cluster controller shrinks the ISR set down to only one broker (the Leader itself).
  • In this degraded state, acks=all requires confirmation from only that single Leader, effectively collapsing the durability guarantee down to acks=1!
  • If that lone surviving Leader suffers a power loss or hardware failure before the other brokers recover, all acknowledged records are permanently destroyed.

The Golden Production Triad:

  1. replication.factor = 3
  2. min.insync.replicas = 2
  3. acks = all
When $ \text{ISR} < \text{min.insync.replicas}$, the Leader immediately rejects the write with a NotEnoughReplicasException, safeguarding consistency over availability.

Comparison of ACKS Modes

| Parameter | Acknowledged By | Latency | Throughput | Data Loss Risk | Retries Supported | Typical Use Case | | :— | :— | :— | :— | :— | :— | :— | | acks=0 | None (Local socket buffer) | < 1 ms | Maximum | Very High | ❌ No (Blind fire) | High-volume telemetry, clickstream, IoT sensor pings | | acks=1 | Leader Replica only | 2–5 ms | High | Moderate (Leader crashes prior to sync) | ✅ Yes | System logs, cache invalidation events | | acks=all | Leader + All ISR members | 5–20 ms | High (with batching) | Zero (with min.insync.replicas >= 2) | ✅ Yes | Financial transactions, orders, billing, state events |

🔴 Senior Level

Broker Internal Architecture: DelayedProduce and Purgatory

When a partition Leader receives a ProduceRequest configured with acks=all, it does not block the active I/O network thread (KafkaRequestHandler):

  1. Local Append: The Leader calls Log.appendAsLeader(), writing the batch to the local OS PageCache.
  2. Enqueue in Purgatory: A DelayedProduce tracking object is registered in the broker’s DelayedOperationPurgatory (an internal hierarchical timing wheel data structure).
  3. Follower Replication: Follower replicas issue FetchRequest RPCs. As their fetch offsets (LEO) reach the newly appended batch, the Leader updates its in-memory replication tracker.
  4. Advancing High Watermark: Once every member of the active ISR has fetched up to the batch offset, the Leader advances its High Watermark (HW).
  5. Callback Completion: The Purgatory triggers the completion callback, compiles the ProduceResponse, and returns RecordMetadata (offset and timestamp) to the waiting producer client.
Producer                     Broker (Leader)                   Follower (ISR)
   |                                |                                 |
   |--- ProduceRequest (acks=all) ->|                                 |
   |                                |-- 1. Log.appendAsLeader()       |
   |                                |-- 2. Register DelayedProduce    |
   |                                |      in Purgatory               |
   |                                |                                 |
   |                                |<--- FetchRequest (LEO update) --|
   |                                |--- FetchResponse -------------->|
   |                                |                                 |
   |                                |-- 3. HW advanced to match LEO   |
   |                                |-- 4. Complete DelayedProduce    |
   |                                |      and release from Purgatory |
   |<-- ProduceResponse (Success) --|                                 |

Interaction Between ACKS, Idempotence, and In-Flight Requests

To achieve end-to-end delivery guarantees without duplicate records during retries:

  • enable.idempotence = true mandates:
    • acks = all (or -1)
    • retries > 0
    • max.in.flight.requests.per.connection <= 5
  • The producer assigns a monotonic SequenceNumber to each batch per ProducerId (PID).
  • If the broker successfully commits the batch but the network ACK is dropped, the producer retries. Upon re-receiving the batch, the Leader detects duplicate sequence numbers, replies with a successful ACK, and silently skips re-appending the duplicate to disk.

4 Tricky Questions

1. Why can the combination of acks=all and replication.factor=3 without min.insync.replicas still cause data loss?

Answer:

  • By default or through misconfiguration, topics may be created with min.insync.replicas = 1.
  • If two follower brokers in a 3-replica topic crash or experience network partitions, the cluster controller shrinks the ISR to just the Leader (ISR = [Leader]).
  • A producer sending records with acks=all asks: “Did all current members of the active ISR acknowledge?”. Since the ISR contains only the Leader, the write is confirmed immediately with no replication.
  • If that Leader broker experiences a subsequent power failure or kernel panic before the followers recover, the newly written data is permanently lost.
  • Setting min.insync.replicas = 2 enforces that the Leader must reject writes with NotEnoughReplicasException whenever the ISR size is less than 2, guaranteeing resilience against single-broker loss.

2. Can a producer configured with acks=0 exhaust memory and crash with an OutOfMemoryError or block?

Answer:

  • Yes, it can.
  • While acks=0 prevents the client from waiting for broker network responses, calls to producer.send() do not bypass client memory buffers. Records must still pass into the internal RecordAccumulator buffer pool before being transmitted by the background Sender I/O thread.
  • If the application produces records faster than the network socket or network interface can physically flush them, the internal buffer (buffer.memory, default 32 MB) will fill completely.
  • Once full, subsequent producer.send() calls block for up to max.block.ms (default: 60,000 ms / 60 seconds). If memory cannot be reclaimed, the producer throws a TimeoutException. If the calling application continually holds unmanaged references to these failed records in the heap, it will ultimately trigger an OutOfMemoryError.

3. A broker suffers a prolonged Stop-the-World GC pause. How does this affect producers configured with acks=all?

Answer:

  • If an In-Sync Follower pauses:
    • The Leader waits for the follower to fetch the pending batch, holding the producer’s request inside the DelayedProduce Purgatory.
    • Producer latency spikes significantly, bounded by request.timeout.ms (default: 30s).
    • After replica.lag.time.max.ms (default: 30s) elapses without a fetch, the Leader evicts the lagging follower from the ISR (ISR Shrink).
    • Once the ISR shrinks, the Leader re-evaluates the pending request. If the remaining ISR satisfies min.insync.replicas, the Leader advances the High Watermark and immediately sends a successful response to the producer.
  • If the Leader itself pauses:
    • The producer blocks until request.timeout.ms expires, throws a TimeoutException, and executes an automated retry against cluster metadata.

4. What is the difference between NotEnoughReplicasException and NotEnoughReplicasAfterAppendException?

Answer:

  • NotEnoughReplicasException: Returned before any disk append occurs. The Leader inspected the current ISR size upon receiving the ProduceRequest, observed that $ \text{ISR} < \text{min.insync.replicas}$, and immediately rejected the request. The Leader’s local log remains completely clean and unmodified.
  • NotEnoughReplicasAfterAppendException: The Leader inspected the ISR at request arrival and verified that $ \text{ISR} \ge \text{min.insync.replicas}$. It successfully appended the batch to its local disk log. However, while waiting in Purgatory for followers to fetch the batch, one or more followers timed out, shrinking the ISR below min.insync.replicas.
  • The message exists in the Leader’s log but cannot be safely acknowledged. When the producer retries with idempotence enabled, the Leader deduplicates the sequence number without appending twice.

🎯 Interview Cheat Sheet

Core Concepts

  • acks=0: Fire-and-forget; highest throughput, lowest latency, zero delivery guarantee, no retries.
  • acks=1: Leader-only acknowledgment; balanced latency, vulnerable to un-replicated Leader crashes.
  • acks=all (or -1): Quorum acknowledgment across all active ISR members; required for enterprise zero data loss.
  • Kafka 3.0 Default Shift: Defaults to acks=all with enable.idempotence=true for out-of-the-box durability.
  • The Durability Standard: replication.factor = 3 + min.insync.replicas = 2 + acks = all.

Mode Comparison Matrix

| Parameter | acks=0 | acks=1 | acks=all | | :— | :— | :— | :— | | Acknowledged By | Local socket buffer | Leader Replica only | Leader + All active ISR members | | Delivery Guarantee | At-most-once | At-least-once (with risk) | At-least-once / Exactly-once (with idempotence) | | Data Loss on Leader Crash | Highly Probable | Possible | Eliminated (with ISR $\ge 2$) | | Default in Kafka Version | Kafka 0.8.x | Kafka 0.9.x – 2.8.x | Kafka 3.0+ |

Red Flags to Avoid

  • ❌ “With acks=all, the producer communicates directly with all brokers in parallel.” (The producer communicates exclusively with the partition Leader; the Leader manages replication).
  • ❌ “Setting acks=all completely eliminates duplicate records.” (acks=all guarantees durability; duplicate prevention requires enable.idempotence=true).
  • ❌ “Messages published with acks=all are immediately readable by consumers.” (Consumers can only read messages committed up to the High Watermark after ISR replication completes).