What is Topic in Kafka

A Topic in Apache Kafka is a named logical category or stream that acts as a distributed, persistent, immutable append-only commit log. Producers publish event records to the ta...

🟢 Junior Level

30-Second Summary

A Topic in Apache Kafka is a named logical category or stream that acts as a distributed, persistent, immutable append-only commit log. Producers publish event records to the tail of the log, and consumers read records sequentially from it.

The fundamental distinction between a Kafka topic and a traditional message queue (RabbitMQ, ActiveMQ):

Traditional Queues: A message is deleted by the broker immediately once a consumer acknowledges processing (destructive read).
Kafka Topics: Records are NOT deleted upon reading. They are persisted to disk according to a configurable retention policy (e.g., 7 days or 100 GB). Multiple independent microservice consumer groups can read the exact same stream in parallel at their own processing speeds, and even rewind offsets to replay historical events from any point in time.

Real-World Analogy

A Topic is like a daily printed newspaper kept in a public library’s archive room:

The publisher (Producer) prints daily editions and appends them to the archive binder.
Library patrons (Consumers) visit the reading room. Patron A reads today’s fresh front page, Patron B reviews last Tuesday’s business section, and Patron C transcribes historical articles from the entire past year.
One patron reading an article does not shred or destroy the newspaper pages — the volume remains on the shelf for all other visitors until the librarian purges outdated issues according to the institution’s archive retention schedule.

Basic Spring Boot 3 Example (Producer & Consumers)

@Service
public class OrderEventService {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public OrderEventService(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    // Publish an order creation event to the "orders" topic partitioned by orderId
    public void publishOrderCreated(String orderId, String payload) {
        kafkaTemplate.send("orders", orderId, payload);
    }

    // Independent consumer group for Billing Service
    @KafkaListener(topics = "orders", groupId = "billing-service-group")
    public void consumeForBilling(String message) {
        // Process customer credit card payment
    }

    // Independent consumer group for Notifications Service (reads same topic concurrently)
    @KafkaListener(topics = "orders", groupId = "notifications-service-group")
    public void consumeForNotifications(String message) {
        // Dispatch SMS / email alert to buyer
    }
}

# Create a topic via Kafka CLI: 3 partitions for parallelism, replication factor of 3
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 3

🟡 Middle Level

Logical vs. Physical Topic Architecture

A topic is a logical organizational entity. Physically across cluster storage nodes, a topic is horizontally split into Partitions:

                             [ Logical Topic: orders ]
                                         │
        ┌────────────────────────────────┼────────────────────────────────┐
        ▼                                ▼                                ▼
  [ Partition 0 ]                  [ Partition 1 ]                  [ Partition 2 ]
  (Broker 1, Leader)               (Broker 2, Leader)               (Broker 3, Leader)
        │                                │                                │
  Files on Disk:                   Files on Disk:                   Files on Disk:
  orders-0/                        orders-1/                        orders-2/
   ├── 00000.log (Payload)          ├── 00000.log                    ├── 00000.log
   ├── 00000.index (Offset index)   ├── 00000.index                  ├── 00000.index
   └── 00000.timeindex (Timestamp)  └── 00000.timeindex              └── 00000.timeindex

Filesystem Directory: On the broker disk, each partition maps to an independent folder named <topic_name>-<partition_id>.
Log Segments (.log): Records are appended to active segment files. When an active segment reaches segment.bytes (default: 1 GB) or segment.ms (default: 7 days), it is sealed, and a new active segment is rolled.
Sparse Indexes (.index, .timeindex): Memory-mapped binary lookup tables that map logical message offsets and timestamps to exact physical byte positions on disk, enabling $O(1)$ random lookups without scanning gigabyte-sized log files.

Topic Cleanup Policies: `delete` vs. `compact`

Policy	Mechanism	Standard Production Use Case
`delete` (default)	Old segments are purged when they exceed age (`retention.ms`) or total partition size (`retention.bytes`).	Temporal event streams: audit trails, telemetry metrics, financial clickstreams.
`compact`	Background `LogCleaner` threads discard older records sharing the same key, retaining only the latest state per key.	Changelogs, user profile state, account balances, Kafka Streams `KTable` backings.

Message Ordering Guarantees

Within a Single Partition: Kafka guarantees strict, deterministic FIFO (First-In, First-Out) ordering. Every message is assigned a monotonically increasing sequential 64-bit integer: the Offset.
Across the Entire Topic (Multiple Partitions): There is NO global ordering guarantee. Partitions are consumed independently and concurrently across consumer threads.
Architectural Takeaway: To preserve strict processing order for related business entities (e.g., state transitions of order-123), the producer must specify a consistent Message Key. All records sharing that key will hash to the exact same partition.

🔴 Senior Level

Zero-Copy and OS Internals: High-Throughput Disk I/O

A frequent interview question: “How does Kafka process millions of messages per second if it writes every record to persistent disk storage?”

Kafka achieves near-memory speeds through three operating system optimizations:

Sequential Disk I/O: Appending to a file (Append-Only) avoids random disk arm seek times. Sequential writes to modern NVMe SSDs hit 3–5 GB/s, matching or exceeding random RAM access speeds.
Linux Page Cache: Kafka bypasses internal Java Heap caching. Records are written straight to OS Page Cache. This completely avoids Java Garbage Collection (GC) pauses and JVM object header memory bloat.

Zero-Copy Network Transfers (sendfile Syscall): When consumers read records, the Linux kernel transfers data directly from the filesystem Page Cache into the network interface card (NIC) DMA buffer:

Traditional Approach: Disk ──► OS PageCache ──► JVM Heap ──► Socket Buffer ──► NIC (4 context switches, 4 copies)
Zero-Copy (sendfile):  Disk ──► OS PageCache ────────────────────────────────► NIC Buffer (2 context switches, 0 CPU copies)

Log Compaction, Tombstones, and the “Data Resurrection” Risk

Under cleanup.policy=compact:

To delete an entity (userId=42), the producer publishes a Tombstone: a message with key="42" and payload value=null.
The background LogCleaner deduplicates older versions and temporarily retains the Tombstone marker.
The Data Resurrection Trap: The configuration delete.retention.ms (default: 24 hours) dictates how long Tombstones persist before physical eviction. If a consumer goes offline or lags for longer than delete.retention.ms, it will never process the Tombstone and will restore obsolete state on its next startup, resurrecting the deleted user in local caches!

High-Durability Contract: The Durability Triad

To prevent data loss in mission-critical banking and trading topics, enforce this configuration triad:

replication.factor = 3: 3 physical replica copies across independent hardware racks.
min.insync.replicas = 2: At least 2 synchronized replicas must commit the write before success is declared.
acks = all (or -1): Producer waits for acknowledgment from all in-sync replicas (ISR).

Edge Case Resolution: If 2 out of 3 brokers crash in a cluster, only the leader remains alive.

A producer sending with acks=all attempts a write.
The leader inspects its ISR pool and observes ISR count = 1 < min.insync.replicas: 2.
The leader rejects the write and throws a NotEnoughReplicasException. This fails fast, protecting uncommitted data from being wiped upon cluster recovery.

4 Tricky Questions

1. Why does Apache Kafka strictly prohibit DECREASING the number of partitions on an existing topic?

Answer:

Records inside a partition constitute an immutable, append-only sequential log indexed strictly by monotonic offsets.
If Kafka permitted shrinking partition counts (e.g., from 4 down to 2), it would face an intractable architectural dilemma: what to do with records residing in partitions 2 and 3?
- Merging them into partitions 0 and 1 is impossible: offset sequences would collide, interleaving records, violating monotonic ordering guarantees, and invalidating index files.
- Deleting them causes unacceptable catastrophic data loss.
Consequently, kafka-topics.sh --alter supports only increasing partition counts. To reduce partitions, you must provision a new topic and migrate data via Kafka Connect or MirrorMaker.

2. What happens to message ordering if an administrator increases partition count from 3 to 6 on an active topic using keyed messages?

Answer: Message ordering per business key will be permanently violated. By default, the Kafka producer maps keyed records to partitions using the Murmur2 hash modulo formula: $\text{partition} = \text{murmur2}(\text{key}) \pmod{\text{totalPartitions}}$

With totalPartitions = 3, all events for order-99 reliably mapped to partition 1.
The instant partition count increases to 6, the modulo divisor changes: subsequent records for order-99 will now hash to partition 4!
Because partitions 1 and 4 are consumed concurrently by different consumers, an OrderCancelled event in partition 4 can be processed before the earlier OrderCreated event in partition 1.
Rule: Partition counts for keyed topics must be carefully capacity-planned upfront to avoid mid-lifecycle alterations.

3. Why does Kafka intentionally avoid caching messages in the Java Virtual Machine Heap (JVM Heap) and instead rely entirely on the Linux Page Cache?

Answer: This architectural design provides three decisive advantages:

Elimination of Java Object Memory Bloat: Every JVM object includes object header metadata (Mark Word + Klass Word) and 8-byte address padding. A 100-byte message can easily consume 300–400 bytes in heap memory. Page Cache stores raw, compacted binary byte buffers directly as received over the wire.
Elimination of GC Stop-The-World (STW) Pauses: Caching tens of gigabytes of messages in JVM heap would trigger massive Garbage Collection sweeps, freezing broker threads for multiple seconds.
Cache Survivability Across Restarts: If the Kafka JVM crashes, runs out of heap, or restarts during an update, the Java heap is wiped. However, the host operating system’s Page Cache remains warm in physical RAM, enabling the restarted broker to instantly serve consumers at peak throughput.

4. What is a “Tombstone” in a compacted topic, and what operational risk arises if `delete.retention.ms` is set too low?