📨 Section 15 · Question #16

What is Replication in Kafka

The number of partition copies is configured per topic via the Replication Factor (RF):

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

Replication in Apache Kafka is the architectural mechanism of copying and maintaining identical partition logs across multiple physical broker nodes to ensure High Availability (HA), Fault Tolerance, and Zero Data Loss.

The number of partition copies is configured per topic via the Replication Factor (RF):

  • Leader Replica: Exactly one replica is elected as the partition Leader. By default, it accepts all incoming write requests from producers and serves read requests to consumers.
  • Follower Replicas: All other replicas passively pull and replicate the Leader’s commit log sequentially via internal FetchRequest calls, maintaining exact byte-for-byte mirrors on their respective disks.
  • Automated Failover: If the broker hosting the Leader crashes or becomes unreachable, the cluster controller instantly elects a new Leader from the healthy followers with zero human intervention or data loss.

Real-World Analogy

Imagine a Chief Notary and Deputy Notaries across separate city offices:

  • Leader: The Chief Notary who personally notarizes all incoming contracts (messages) into the original master ledger.
  • Followers: Deputy notaries in separate office buildings who constantly fetch carbon copies of newly certified pages from the Chief and paste them into their duplicate ledgers.
  • If the Chief’s office suffers a power outage or fire, the deputy with the most up-to-date duplicate ledger is immediately appointed as the new Chief Notary.

Creating a Replicated Topic

# Create an orders topic with 3 partitions and a replication factor of 3
kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 3 \
  --config min.insync.replicas=2

🟡 Middle Level

ISR (In-Sync Replicas) Dynamics

The Leader broker continuously tracks the health and replication progress of all followers using the ISR (In-Sync Replicas) set:

  • A follower is considered in-sync if it continuously fetches messages from the Leader and its fetch lag does not exceed replica.lag.time.max.ms (default: 30,000 ms / 30 seconds).
  • If a follower stalls due to long GC pauses, network degradation, or saturated disk I/O, the Leader evicts it from the ISR set.
  • Once the lagging follower catches up to the Leader’s Log End Offset (LEO), it is automatically reinstated into the ISR set.
Cluster with Replication Factor = 3:

[ Broker 1 (Leader) ] ───(Fetch)───► [ Broker 2 (Follower) ]  -> Caught up (Lag < 30s) -> In ISR ✅
        │
        └────────────────(Fetch)───► [ Broker 3 (Follower) ]  -> Lag > 30s -> EVICTED from ISR ❌

Current ISR Set: [Broker 1, Broker 2]

The Production Durability Triad

To guarantee 100% data durability and eliminate data loss across broker crashes, production environments must enforce the Durability Triad:

  1. replication.factor = 3 (Topic level) — 3 independent physical copies distributed across distinct nodes.
  2. min.insync.replicas = 2 (Topic/Broker level) — The minimum number of acknowledged ISR replicas required to confirm a write.
  3. acks = all (or -1) (Producer level) — The producer blocks until the Leader and all current ISR members acknowledge receipt.
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092,broker3:9092");
// Guarantee all ISR replicas acknowledge before returning success
props.put(ProducerConfig.ACKS_CONFIG, "all");
// Enable idempotence to prevent duplicate records during network retries
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

🔴 Senior Level

Internal Replication Landmarks: LEO, HW, and Leader Epoch

Kafka partition replication is managed through three fundamental pointer offsets:

  Offset:   0     1     2     3     4     5     6     7     8
          ┌───┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐
  Log:    │ m0│ m1  │ m2  │ m3  │ m4  │ m5  │ m6  │ m7  │     │
          └───┴─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘
                                      ▲                 ▲
                                      │                 │
                                High Watermark (HW)   Log End Offset (LEO)
  1. Log End Offset (LEO): The offset of the next record to be written into the local disk log of that specific replica. The Leader and each follower maintain independent LEOs.
  2. High Watermark (HW): The highest offset that has been replicated to all replicas currently in the ISR.
    • Consumer Boundary Rule: Standard consumers are only permitted to read records up to the High Watermark! Uncommitted messages beyond HW remain invisible to prevent phantom reads.
  3. Leader Epoch (KIP-101): A monotonically increasing epoch number incremented upon every leader election. It solves log divergence and silent data loss caused by older High Watermark truncation edge cases.

Reading from Nearest Replicas (Rack Awareness / KIP-392)

Historically, consumers were strictly restricted to reading from partition Leaders. In multi-availability-zone cloud environments (e.g., AWS multi-AZ, GCP regions), this created massive cross-AZ network egress bandwidth charges.

Introduced in Kafka 2.4 (KIP-392), consumers can read directly from local Follower replicas within the same rack or cloud availability zone:

# On the Kafka Broker:
broker.rack=eu-west-1a

# On the Kafka Consumer client:
client.rack=eu-west-1a
  • The follower replica serves the consumer directly, but strictly up to its own local High Watermark, preserving consistency while eliminating cross-AZ latency and costs.

4 Tricky Questions

1. Why is the combination of replication.factor = 3 and acks = all completely ineffective at preventing data loss if min.insync.replicas = 1?

Answer:

  • A producer with acks = all waits for acknowledgments from all replicas currently present in the active ISR set, not the total configured replication.factor.
  • If two follower brokers crash or experience network partitions, the Leader broker evicts them from the ISR, reducing the set to only itself (ISR = [Broker 1]).
  • Because min.insync.replicas = 1, the Leader evaluates that the minimum quorum threshold is satisfied ($1 \ge 1$), commits the message locally, and replies with a success ACK to the producer.
  • If Broker 1’s disk or node fails in the next millisecond, the message is permanently lost, despite the producer requesting acks = all.
  • Conclusion: Only when min.insync.replicas = 2 will the Leader reject the write with a NotEnoughReplicasException, preventing un-replicated commits.

2. How did Leader Epoch (KIP-101) eliminate log truncation and phantom data loss caused by the legacy High Watermark protocol?

Answer:

  • Under the legacy protocol, follower replicas updated their High Watermark only on subsequent FetchResponse round-trips from the Leader.
  • If a follower wrote a record locally but crashed before receiving the updated HW, on restart it automatically truncated its local log down to its last checkpointed HW (treating records past HW as uncommitted).
  • If the Leader crashed at the same time and this follower was elected as the new Leader, previously acknowledged records were permanently dropped from the cluster.
  • Leader Epoch Solution: Replicas maintain an epoch checkpoint file (leader-epoch-checkpoint). When a restarted follower reconnects, it issues an OffsetForLeaderEpochRequest to the new Leader (“What is your start offset for Epoch $N$?”). Instead of blindly truncating to HW, the replica truncates only if the logs actually diverged under different epochs, preventing accidental loss of valid data.

3. What is the difference between NotEnoughReplicasException and NotEnoughReplicasAfterAppendException?

Answer:

  • NotEnoughReplicasException: The Leader verified the ISR size prior to appending the record to its local log. Detecting that $ \text{ISR} < \text{min.insync.replicas}$, it immediately rejected the producer’s request without writing to disk. The Leader’s log remains unaffected.
  • NotEnoughReplicasAfterAppendException: The ISR met the quorum requirement when the write arrived, and the Leader successfully appended the record to its local disk segment. However, while waiting for followers to fetch the record, one or more followers dropped out of ISR, dropping the quorum below min.insync.replicas.
  • Both exceptions trigger producer retries, but when idempotence is enabled (enable.idempotence=true), duplicate writes on the Leader are transparently deduplicated by sequence number.

4. What is unclean.leader.election.enable, and why is it strictly set to false by default in modern Kafka?

Answer:

  • If the Leader broker fails and no surviving follower is currently in the ISR (all followers lagged or crashed):
    • With unclean.leader.election.enable = false: The partition becomes completely unavailable for reads and writes until a member of the ISR recovers. The system chooses Consistency over Availability (CP in CAP theorem).
    • With unclean.leader.election.enable = true: The cluster permits an out-of-sync replica (non-ISR follower) to be elected Leader.
  • The critical risk: An out-of-sync replica is missing recent messages. All writes committed on the previous Leader that never reached this follower are wiped out. Furthermore, when other replicas reconnect, they truncate their logs to match the new out-of-sync leader, causing silent, irreversible data corruption across the system.

🎯 Interview Cheat Sheet

Core Concepts

  • Definition: Replicating partition logs across multiple physical brokers to ensure fault tolerance.
  • Replication Roles: Exactly one Leader (handles writes and standard reads) and $N - 1$ Followers (passive log replicators).
  • ISR (In-Sync Replicas): Dynamic set of replicas whose fetch delay from the Leader does not exceed replica.lag.time.max.ms.
  • Production Standard: replication.factor = 3, min.insync.replicas = 2, acks = all.
  • High Watermark (HW): The highest offset replicated across all ISR members; marks the upper visibility boundary for consumers.

Replication Availability and Durability Matrix

| Scenario | Write Availability | Read Availability | Data Loss Risk | | :— | :— | :— | :— | | ISR = 3, min.insync = 2 | ✅ Full | ✅ Full | ❌ Zero | | ISR = 2, min.insync = 2 | ✅ Full | ✅ Full | ❌ Zero | | ISR = 1, min.insync = 2 | ❌ NotEnoughReplicasException | ✅ Full (up to HW) | ❌ Zero | | ISR = 1, min.insync = 1 | ✅ Full | ✅ Full | ⚠️ Critical risk if Leader crashes |

Red Flags to Avoid

  • ❌ “Follower replicas accept producer write requests in parallel to scale cluster throughput.” (Only the Leader accepts writes; followers replicate asynchronously).
  • ❌ “Setting acks = all guarantees zero data loss even if min.insync.replicas = 1.” (If followers drop out, the Leader writes alone and single-broker failure causes loss).
  • ❌ “Consumers immediately read messages as soon as the Leader appends them to disk.” (Consumers are strictly restricted to reading offsets $\le \text{High Watermark}$).