Junior Level

Auto Commit

The KafkaConsumer client automatically sends commit requests every auto.commit.interval.ms. This is client-side logic, not broker-side — the broker doesn’t initiate commits.

props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "5000");  // every 5 seconds

Pros:

• Simple, no need to call manually
• Kafka handles commits itself

Cons:

• Possible losses or duplicates
• Commits even if not processed

Manual Commit

You yourself decide when to save the offset.

props.put("enable.auto.commit", "false");
consumer.commitSync();  // when you want

Pros:

• Full control
• Commit after processing

Cons:

• Need to call manually
• More code

Comparison

// Auto commit — simple
props.put("enable.auto.commit", "true");
consumer.subscribe(List.of("orders"));
// Kafka commits every 5 seconds automatically

// Manual commit — reliable
props.put("enable.auto.commit", "false");
while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (var record : records) {
        process(record);
    }
    consumer.commitSync();  // after processing
}

---

Middle Level

Auto Commit — details

How it works:
1. Consumer poll → Kafka automatically commits
2. Every auto.commit.interval.ms
3. Commits the last received offset (not the processed one!)

Problems:
- Commits before processing → loss on crash
- Fixed interval — independent of processing
- No error control

Manual Commit — options

Sync Commit:

// Waits for broker confirmation
consumer.commitSync();
// Reliable, but slower

Async Commit:

// Doesn't wait for confirmation
consumer.commitAsync((offsets, exception) -> {
    if (exception != null) {
        log.error("Commit failed", exception);
    }
});
// Async commit sent the request and didn't wait. If the broker crashed before writing — offset not saved.
// On restart, the consumer will read the same messages again (duplicates, not data loss).

Common mistakes

1. Auto commit + long processing:

   Commit every 5 sec → processing takes 10 sec → message loss
   

2. Manual commit without error handling:

   consumer.commitAsync();  // error → nobody knows
   

3. Mixed commit strategies:

   // Auto and manual commit simultaneously
   // → Unpredictable behavior
   

When to Use What

Strategy	When to use
Auto commit	Prototypes, tests, non-critical data
Manual sync commit	Production, critical data
Manual async commit	High-throughput, tolerant to duplicates

---

Senior Level

Internal Implementation

Auto Commit:

Implemented in KafkaConsumer:
- On each poll() checks if interval has passed
- If yes → sends commit request
- Doesn't wait for response (fire and forget)

Manual Commit:

commitSync():
  - Sends request → waits for response
  - Retry on transient errors
  - Throws exception on failure

commitAsync():
  - Sends request → doesn't wait
  - Callback on completion
  - No retry (caller responsibility)

Auto Commit Problems — Deep Dive

1. Offset Lag:

Auto commit commits the received offset, not the processed one

Timeline:
  T=0: Poll received offset 100
  T=2: Auto commit offset 100
  T=5: Started processing offset 100
  T=8: Crashed

Result: offset 100 lost (committed, not processed)

2. Commit Interval Mismatch:

Fast processing + long commit interval:
  Processed 1000 messages → waits 5 seconds → commit
  If crashes → will process 1000 messages again

Slow processing + short commit interval:
  Commits unprocessed offsets → data loss

Production Patterns

1. Manual Sync Commit after batch:

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    try {
        for (var record : records) {
            process(record);
        }
        consumer.commitSync();  // only after processing all
    } catch (Exception e) {
        log.error("Processing failed, offsets not committed", e);
        // Don't commit → messages will be read again
    }
}

2. Async Commit with Fallback:

consumer.commitAsync((offsets, exception) -> {
    if (exception != null) {
        // Fallback to sync commit
        try {
            consumer.commitSync(offsets);
        } catch (Exception e) {
            log.error("Both async and sync commit failed", e);
        }
    }
});

3. Periodic Commit:

// Commit every N seconds regardless of batches
ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
scheduler.scheduleAtFixedRate(() -> {
    consumer.commitAsync();
}, 0, 5, TimeUnit.SECONDS);

Performance Comparison

Latency overhead (per commit):
  Auto commit:     ~0ms (fire and forget)
  Async commit:    ~1ms (callback)
  Sync commit:     ~10-50ms (blocks)

Reliability:
  Auto commit:     Low (no error handling)
  Async commit:    Medium (callback, no retry)
  Sync commit:     High (retry, confirmation)

Monitoring

Key metrics:

kafka.consumer:commit-latency-avg
kafka.consumer:commit-rate
kafka.consumer:failed-commit-rate
kafka.consumer:last-commit-latency

Best Practices

✅ Manual commit for production
✅ CommitSync for critical data
✅ CommitAsync for high-throughput
✅ Error handling for all commits
✅ Monitor commit health

❌ Auto commit for production
❌ Auto commit for critical data
❌ Without commit error handling
❌ Mixed commit strategies
❌ Ignoring commit latency

Architectural decisions

1. Manual sync commit — standard for production
2. Async commit with fallback — performance/reliability balance
3. Auto commit is not recommended for production with critical data. Acceptable for: metrics, logs, real-time analytics with acceptable loss.
4. Error handling is mandatory — retry or alert

Summary for Senior

• Auto commit commits the received, not processed offset
• Manual commit provides control over reliability
• Sync commit for reliability, async for throughput
• Error handling is mandatory for production
• Monitoring commit latency is critical for health checks

---

🎯 Interview Cheat Sheet

Must know:

• Auto commit: every auto.commit.interval.ms, commits the received (not processed!) offset
• Manual commit: full control, commit AFTER processing — at-least-once guarantee
• Sync commit: blocks until confirmed (10-50ms), reliable; Async: non-blocking, callback
• Auto commit problems: offset lag (commits before processing), interval mismatch
• Async commit with fallback: on error → sync commit (single retry attempt)
• Manual commit — standard for production; auto commit — prototypes/tests only
• Mixed commit strategies = unpredictable behavior

Common follow-up questions:

• Why is auto commit dangerous? — Commits the received offset, not processed → on crash, data is lost.
• Which is better, sync or async? — Sync for reliability, async for throughput with fallback on error.
• How does async commit fallback work? — On callback error → sync commit → if it also fails → log error.
• Is auto commit broker-side? — No, it’s client-side logic in KafkaConsumer, not broker.

Red flags (DO NOT say):

• “Auto commit is a good choice for production” — commits before processing
• “Async commit without error handling is fine” — offset loss
• “Auto commit is initiated by the broker” — it’s client-side logic
• “You can mix auto and manual commit” — unpredictable behavior

Related topics:

• [[13. How does offset commit work]]
• [[12. What is offset in Kafka]]
• [[15. What is rebalancing and when does it happen]]
• [[5. What is Consumer Group]]