Kafka
30 interview questions and answers in the Kafka section.
Questions in this section
- What is topic in Kafka
- What is partition and why is it needed
- How is data distributed across partitions
- What is message key and how does it affect partitioning
- What is Consumer Group
- How does consumer balancing work in a group
- Can you have more consumers than partitions
- What happens when you add a new consumer to the group
- What message delivery guarantees does Kafka provide
- What is the difference between at-most-once, at-least-once and exactly-once
- How to configure exactly-once semantics
- What is offset in Kafka
- How does offset commit work
- What is the difference between auto commit and manual commit
- What is rebalancing and when does it happen
- What is replication in Kafka
- What are leader and follower replicas
- What is ISR (In-Sync Replicas)
- How does Kafka ensure fault tolerance
- What is producer acknowledgment and what modes exist (acks=0,1,all)?
- What is batch in Kafka producer
- How does message compression work
- What is idempotent producer
- How to handle errors when reading messages in Kafka
- What is DLQ (Dead Letter Queue) in Kafka
- How to monitor consumer lag in Kafka
- What is retention policy in Kafka
- How are old messages deleted from a Kafka topic
- Can you read messages from a specific partition in Kafka
- How to implement message filtering on consumer side in Kafka
Study navigator
30 interview preparation questions for Middle/Senior Java Developer.
📋 All Questions
🗺️ Topic Dependency Map
┌──────────────────────────────────────────┐
│ KAFKA BASICS (1-4) │
│ 1. Topic │
│ 2. Partition │
│ 3. Data Distribution │
│ 4. Message Key │
└──────────────────┬───────────────────────┘
│
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌────────────────────┐
│ CONSUMER │ │ DELIVERY │ │ PRODUCER │
│ GROUPS (5-8) │ │ GUARANTEES │ │ (20-23) │
│ 5. Consumer │ │ (9-11) │ │ 20. acks=0,1,all │
│ Group │ │ 9. Guarantees │ │ 21. Batch │
│ 6. Balancing │ │ 10. At-most/ │ │ 22. Compression │
│ 7. More cons. │ │ least/ │ │ 23. Idempotent │
│ 8. Adding new │ │ exactly │ │ producer │
│ consumer │ │ 11. Configure │ │ │
└───────┬───────┘ └───────┬───────┘ └────────┬───────────┘
│ │ │
└────────────────────────┼────────────────────────┘
▼
┌──────────────────────────────────────────┐
│ OFFSET AND REBALANCING (12-15) │
│ 12. Offset │
│ 13. Commit offset │
│ 14. Auto vs Manual commit │
│ 15. Rebalancing │
└──────────────────────────────────────────┘
│
┌────────────────────────┼────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌────────────────────┐
│ REPLICATION │ │ ERRORS AND │ │ RETENTION AND │
│ AND FA (16-19) │ │ DLQ (24-26) │ │ FILTERING │
│ 16. Replica. │ │ 24. Error │ │ (27-30) │
│ 17. Leader/ │ │ handling │ │ 27. Retention │
│ Follower │ │ 25. DLQ │ │ 28. Deletion │
│ 18. ISR │ │ 26. Lag monit. │ │ 29. Partition read │
│ 19. Fault │ │ │ │ 30. Filtering │
│ tolerance │ │ │ │ │
└───────────────┘ └───────────────┘ └────────────────────┘
🎯 Recommended Study Order
🟢 Junior Level (weeks 1-2)
| Step | Topic | Files | Goal |
|---|---|---|---|
| 1 | Basics | Q1, Q2, Q3, Q4 | Topic, partition, key, distribution |
| 2 | Consumer Group | Q5, Q6 | Groups, balancing |
| 3 | Offset and commit | Q12, Q13, Q14 | Offset, commit types |
| 4 | Delivery guarantees | Q9, Q10 | At-most, at-least, exactly-once |
| 5 | Producer acks | Q20 | acks=0,1,all |
🟡 Middle Level (weeks 3-4)
| Step | Topic | Files | Goal |
|---|---|---|---|
| 1 | Rebalancing | Q7, Q8, Q15 | When it happens, eager vs cooperative |
| 2 | Replication | Q16, Q17, Q18 | ISR, leader/follower, fault tolerance |
| 3 | Batch & compression | Q21, Q22 | batch.size, linger.ms, lz4/gzip/snappy |
| 4 | Error handling | Q24, Q25 | Retry, DLQ, poison pill |
| 5 | Retention | Q27, Q28 | retention.ms, retention.bytes, cleanup.policy |
| 6 | Monitoring | Q26 | Consumer lag, tools |
🔴 Senior Level (weeks 5-6)
| Step | Topic | Files | Goal |
|---|---|---|---|
| 1 | Exactly-once | Q11 | EOS v1 vs v2, Producer ID/Epoch, Outbox |
| 2 | Idempotent producer | Q23 | PID, Epoch, max.in.flight, deduplication |
| 3 | Fault tolerance deep | Q19 | RF, min.insync.replicas, unclean election |
| 4 | Advanced consumer | Q29, Q30 | assign() vs subscribe(), filtering patterns |
| 5 | ISR deep dive | Q18 | ISR shrink/expand, LEO, HW, leader epoch |
| 6 | Chaos engineering | Q15 (Senior) | Cascading rebalancing, GC pauses, tuning |
🔗 Key Topic Connections
Topic: Kafka Basics
Q1 (Topic) → Q2 (Partition) → Q3 (Distribution) → Q4 (Key)
Key connections:
- Q2 ↔ Q16: Partitions are replicated for fault tolerance
- Q3 ↔ Q20: acks=all affects when a message is considered delivered
- Q4 ↔ Q3: Key determines partition via MurmurHash
Topic: Consumer Groups
Q5 (Consumer Group) → Q6 (Balancing) → Q7 (More consumers)
↓
Q8 (Adding consumer) → Q15 (Rebalancing)
Key connections:
- Q5 ↔ Q12: Each consumer commits offset for each partition
- Q6 ↔ Q15: Balancing is the rebalance itself
- Q7 ↔ Q8: Idle consumers waste resources, adding a new one triggers rebalance
Topic: Offset and Delivery
Q12 (Offset) → Q13 (Commit) → Q14 (Auto vs Manual)
↓
Q9 (Guarantees) → Q10 (At-most/at-least/exactly) → Q11 (Configure exactly-once)
Key connections:
- Q13 ↔ Q9: Commit BEFORE processing = at-most-once, AFTER = at-least-once
- Q14 ↔ Q13: Auto commit = simpler but risk of loss; Manual = control
- Q10 ↔ Q11: Exactly-once requires transactional producer
Topic: Producer
Q20 (acks) → Q21 (Batch) → Q22 (Compression) → Q23 (Idempotent)
Key connections:
- Q20 ↔ Q23: Idempotent producer + acks=all = exactly-once delivery to topic
- Q21 ↔ Q22: Batch + compression = maximum throughput
- Q23 ↔ Q11: Idempotent producer — the foundation of exactly-once
Topic: Fault Tolerance
Q16 (Replication) → Q17 (Leader/Follower) → Q18 (ISR) → Q19 (Fault tolerance)
Key connections:
- Q17 ↔ Q18: Leader is always in ISR, followers drop out if lagging
- Q18 ↔ Q19: ISR shrink → reduced fault tolerance
- Q16 ↔ Q20: acks=all waits for acknowledgment from all ISR replicas
Topic: Operations
Q24 (Error handling) → Q25 (DLQ) → Q26 (Lag monitoring)
↓
Q27 (Retention) → Q28 (Deletion) → Q29 (Partition read) → Q30 (Filtering)
Key connections:
- Q24 ↔ Q25: DLQ — the last resort after retries
- Q25 ↔ Q26: Lag monitoring helps detect issues before DLQ
- Q27 ↔ Q28: Retention policy determines when and how messages are deleted
- Q29 ↔ Q30: assign() for reading specific partitions, filtering on consumer side
🎓 Cheat Sheet: What to Know for Each Level
🟢 Junior
- Topic = logical channel, partition = sub-channel for parallelism
- Consumer Group = mechanism for distributing partitions among consumers
- Offset = position in partition, commit = saving position
- acks=0 (fire-and-forget), acks=1 (leader only), acks=all (all ISR)
- At-most-once, at-least-once, exactly-once — three delivery semantics
🟡 Middle
- Rebalancing: eager (full stop) vs cooperative (partial)
- ISR: replicas that caught up with the leader. Leader is always in ISR
- Batch: batch.size, linger.ms, compression — three throughput levers
- DLQ: queue for messages that couldn’t be processed after N retries
- Retention: retention.ms (time) + retention.bytes (size per partition)
- Sticky partitioning: Kafka 2.4+ replaces round-robin
🔴 Senior
- Exactly-once: EOS v2 (Kafka 2.5+), Producer ID + Epoch, transaction coordinator
- Idempotent producer: deduplication at broker level via PID + sequence
- ISR shrink/expand: monitoring is critical for fault tolerance
- Cascading rebalancing: GC pause → exclude → rebalance → overload → another one falls
- Outbox pattern: for exactly-once Kafka-to-Database
- Leader Epoch: tracks data loss during failover
📝 File Format
Each file contains:
- 🟢 Junior Level — basic understanding, simple analogies, examples
- 🟡 Middle Level — internals, common mistakes, practical examples
- 🔴 Senior Level — deep dive, edge cases, production experience, monitoring
- 🎯 Interview Cheat Sheet — key takeaways, common questions, red flags, related topics