📨 Section 15 · Question #14

What is the Difference Between Auto Commit and Manual Commit

The fundamental difference between Auto Commit and Manual Commit centers on who initiates offset commits and at what moment progress is persisted to the cluster:

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

The fundamental difference between Auto Commit and Manual Commit centers on who initiates offset commits and at what moment progress is persisted to the cluster:

  • Auto Commit (enable.auto.commit = true): The Kafka client library automatically commits offsets periodically based on a timer (auto.commit.interval.ms, default 5 seconds). It commits the offsets of records fetched, regardless of whether downstream business logic has succeeded.
  • Manual Commit (enable.auto.commit = false): The application takes explicit programmatic control. Offsets are committed via commitSync() or commitAsync() strictly after the business transaction has executed and persisted downstream state.

Real-World Analogy

Think of the difference as auto-saving a document in a word processor:

  • Auto Commit: The software presses Ctrl+S every 5 seconds regardless of your document’s state. If you accidentally paste corrupted data or delete a paragraph right before an auto-save, the broken file is saved permanently. If power cuts out mid-edit, state recovery is unpredictable.
  • Manual Commit: You click “Save” only after you have reviewed the text, verified calculations, fixed typos, and confirmed that the final draft is accurate and complete.

Crash Behavior Comparison

Scenario: Application Crashes Unexpectedly During Record Processing

1. With Auto Commit (enable.auto.commit = true):
   Poll Offset 10 ──► Timer fires (Offset 10 committed to broker) ──►
   JVM Crashes 💥 before DB write ──► Service Restarts ──► Reads from Offset 11!
   ❌ Outcome: Record 10 is permanently lost without trace (At-Most-Once).

2. With Manual Commit (enable.auto.commit = false):
   Poll Offset 10 ──► Processing fails / JVM Crashes 💥 ──► NO commit was issued ──►
   Service Restarts ──► Reads from Offset 10 again!
   ✅ Outcome: Record 10 is re-processed safely (At-Least-Once).

🟡 Middle Level

The Polling Thread Invariant: When Does Auto Commit Actually Fire?

A widespread misconception among developers is that auto-commit runs on an independent background daemon thread. This is incorrect!

  • In the official Java KafkaConsumer, there is no background thread for auto-committing. The client follows a single-threaded execution model.
  • Offset evaluation occurs synchronously inside the consumer.poll() method.

  • When poll() is invoked, the client evaluates:

    \[\text{System.currentTimeMillis}() - \text{lastCommitTime} \ge \text{auto.commit.interval.ms}\]
  • If the condition is met, the client sends an asynchronous commit request for the highest offset returned by the previous poll.
  • Direct Consequence: If message processing inside your loop blocks for 30 seconds, no auto-commit can execute until control returns to the next consumer.poll() call!

Manual Commit Implementation in Java

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-fulfillment-group");
// 1. Mandatory: Disable auto-commit
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));

    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
        for (ConsumerRecord<String, String> record : records) {
            processOrder(record.value());
        }

        // 2. Commit offsets strictly after the entire batch is successfully processed
        try {
            consumer.commitSync(); // Or consumer.commitAsync() for higher throughput
        } catch (CommitFailedException e) {
            log.error("Consumer group rebalance revoked our partitions mid-processing!", e);
        }
    }
}

Spring Kafka ContainerProperties.AckMode Options

In Spring Boot, developers configure offset commits using Spring’s container acknowledgement modes:

Spring AckMode Commit Execution Mechanics
RECORD Commits offset immediately after each individual @KafkaListener method execution finishes.
BATCH (default) Commits once after all records returned by the underlying poll() batch have been processed.
TIME Commits when the configured ackTime has elapsed since the last commit.
COUNT Commits when ackCount records have been successfully processed.
MANUAL Application calls Acknowledgment.acknowledge(); commits are aggregated and sent on batch completion.
MANUAL_IMMEDIATE Commits to Kafka immediately upon invoking Acknowledgment.acknowledge().

🔴 Senior Level

System Delivery Semantics: At-Most-Once vs. At-Least-Once vs. EOS

The commit strategy chosen dictates the delivery semantics of your entire distributed system:

  1. At-Most-Once ($0 \le N \le 1$):
    • Characteristic of Auto Commit.
    • Offsets are recorded before downstream persistence is confirmed. If a crash occurs, data is lost, but duplicate writes never happen. Suitable only for high-frequency telemetry, clickstreams, or metrics.
  2. At-Least-Once ($N \ge 1$):
    • Achieved through Manual Commit strictly after persistence.
    • Zero data loss, but duplicate processing can occur upon container crashes. Requires idempotent consumers (e.g., PostgreSQL ON CONFLICT DO NOTHING or unique transaction keys in Redis).
  3. Exactly-Once Semantics (EOS) ($N = 1$):
    • Cannot be achieved using plain commitSync().
    • Requires bundling consumer offset commits and output topic writes into an atomic Two-Phase Commit transaction via KafkaProducer.sendOffsetsToTransaction().

Transactional Offset Commit (Read-Process-Write)

When consuming from Topic A and writing to Topic B, offset commits must be encapsulated within the producer’s transaction:

producer.beginTransaction();
try {
    for (ConsumerRecord<String, String> record : records) {
        String transformed = enrich(record.value());
        producer.send(new ProducerRecord<>("output-topic", record.key(), transformed));
    }

    // Bind consumer offsets to the producer's atomic transaction:
    Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = Map.of(
        new TopicPartition(record.topic(), record.partition()),
        new OffsetAndMetadata(record.offset() + 1)
    );
    
    producer.sendOffsetsToTransaction(offsetsToCommit, consumer.groupMetadata());
    producer.commitTransaction(); // Atomic 2PC: commits both output messages and offsets
} catch (ProducerFencedException | OutOfOrderSequenceException e) {
    producer.abortTransaction();
}

4 Tricky Questions

1. Which thread physically runs Auto Commit, and why won’t auto.commit.interval.ms = 1000 commit offsets after 1 second if your record processing loop takes 20 seconds?

Answer:

  • In the standard Java KafkaConsumer, there is no background daemon thread for auto-committing. The client is designed around a single-threaded event loop.
  • The auto-commit timer check is embedded directly within the implementation of consumer.poll().
  • If a consumer polls a batch and takes 20 seconds processing database updates or external HTTP calls, poll() is never called throughout those 20 seconds.
  • Consequently, no commit is transmitted at the 1-second mark. The offset is committed only when the consumer finishes the 20-second processing cycle and calls consumer.poll() again.

2. How can Auto Commit cause silent message loss (At-Most-Once), even when the developer writes no commit logic?

Answer:

  • Consider this real-world failure progression:
    1. consumer.poll() retrieves 100 messages (offsets 1–100).
    2. Processing the first 10 messages takes 6 seconds (exceeding auto.commit.interval.ms = 5000).
    3. The consumer loop finishes iteration and invokes poll() for the next batch. Inside poll(), the client detects that 5,000 ms elapsed and automatically issues a commit for offset 101.
    4. The second poll() returns offsets 101–200.
    5. During processing of record 105, the application suffers a hardware crash or fatal OOMKilled.
    6. Upon restart, the consumer queries the coordinator and resumes from committed offset 101!
    7. If processing failed on batch 1 before completing, or if batch 2 crashed mid-flight, those unwritten records are permanently lost.

3. Why is it dangerous to manually call consumer.commitSync() inside a Spring @KafkaListener method?

Answer:

  • Spring Kafka’s MessageListenerContainer completely manages the consumer lifecycle, the poll() loop, and thread synchronization.
  • Calling consumer.commitSync() directly inside @KafkaListener:
    1. Thread Race Conditions: Bypasses Spring’s internal state management and conflicts with the configured AckMode.
    2. Duplicate Network Commits: Spring may attempt to commit offsets again at the end of the batch listener loop.
    3. Broken Error Handling: Spring’s error recovery mechanisms (such as DefaultErrorHandler and Dead Letter Topics) rely on re-seeking uncommitted offsets. If application code has already committed the failed record’s offset directly to the broker, Spring cannot replay or redirect the message to a DLT.
  • Production Fix: Use Spring’s Acknowledgment parameter with AckMode.MANUAL or MANUAL_IMMEDIATE.

4. What is the operational difference between AckMode.MANUAL and AckMode.MANUAL_IMMEDIATE in Spring Kafka, and when does it impact performance?

Answer:

  • AckMode.MANUAL: Calling acknowledgment.acknowledge() simply sets an in-memory boolean flag indicating the record is processed. The physical network commit RPC to Kafka is deferred until all records in the current poll() batch have finished processing. This aggregates commits into a single network RPC per batch.
  • AckMode.MANUAL_IMMEDIATE: Calling acknowledge() immediately triggers a network commit RPC to the broker on every single invocation.
  • Performance Impact: If a batch contains 500 records, MANUAL_IMMEDIATE generates 500 distinct synchronous/asynchronous network commit requests per batch, collapsing consumer throughput. It should be reserved exclusively for rare, low-frequency, high-value transactions.

🎯 Interview Cheat Sheet

Core Concepts at a Glance

  • Auto Commit (enable.auto.commit = true): Timer-based, evaluated synchronously inside poll(). Prone to message loss.
  • Manual Commit (enable.auto.commit = false): Application explicitly commits after downstream persistence succeeds.
  • At-Least-Once Delivery: Achieved via Manual Commit + Idempotent Consumer (database unique keys).
  • Spring Kafka Standards: Configure AckMode.MANUAL with Acknowledgment.acknowledge(); never invoke consumer.commitSync() directly inside a listener.
  • Read-Process-Write: Multi-topic pipelines require producer.sendOffsetsToTransaction() for true Exactly-Once guarantees.

Commit Mode Comparison

Feature Auto Commit Manual Commit
Configuration enable.auto.commit = true enable.auto.commit = false
Trigger Point Inside poll() when interval expires Explicitly called by application code
Delivery Semantic At-Most-Once (risk of data loss) At-Least-Once (risk of duplicate replays)
Code Complexity Zero boilerplate Requires error handling and retry logic
Production Suitability ❌ Metrics, IoT telemetry, logs ✅ Financial transactions, orders, billing

Red Flags to Avoid

  • ❌ “Auto-commit runs on an independent background daemon thread every 5 seconds.” (It executes synchronously inside the poll() call).
  • ❌ “Enabling auto-commit is safe for production as long as the service has good unit test coverage.” (Process crashes will still trigger silent data loss).
  • ❌ “Inside a @KafkaListener, I always invoke consumer.commitSync().” (Breaks Spring’s DefaultErrorHandler and AckMode lifecycle).
  • ❌ “Manual commit guarantees Exactly-Once delivery to PostgreSQL automatically.” (It provides At-Least-Once; idempotency keys are required to avoid duplicates).