📨 Section 15 · Question #5

What is Consumer Group

A Consumer Group in Apache Kafka is a logical cooperative cluster of consumer instances (threads or application processes) sharing a common group.id that work together to consum...

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

A Consumer Group in Apache Kafka is a logical cooperative cluster of consumer instances (threads or application processes) sharing a common group.id that work together to consume and process data from one or more topics in parallel.

Core Rules of a Consumer Group:

  1. Exclusive Partition Ownership: Each partition in a topic is assigned to at most one consumer instance within the group at any given time, preventing duplicate processing.
  2. Horizontal Parallelism: If a topic contains 10 partitions and the group has 10 consumers, data is ingested across 10 concurrent streams.
  3. Group Independence (Pub/Sub): Different consumer groups (e.g., group.id = "billing-service" and group.id = "audit-analytics") consume the same topic completely independently. Each group maintains its own committed offsets and receives a full, isolated copy of all topic messages.

Real-World Analogy

Think of a Consumer Group as a crew of logistics workers unloading delivery semi-trailers:

  • The topic is a fleet of 4 semi-trucks (4 partitions).
  • The “Builders” crew (group.id = "builders") consists of 4 workers (consumers).
  • The warehouse supervisor assigns each worker to unload exactly one truck. Two builders never climb into the same truck at the same time—they do not get in each other’s way and never unload the same crate twice.
  • If an independent “Auditors” crew (group.id = "auditors") operates at the facility, they inspect the contents of the exact same 4 trucks on their own schedule without interfering with the builders.

Basic Java 21 / Spring Boot 3 Example

@Service
public class OrderNotificationConsumer {

    // "notifications-group": multiple replicas of this service share topic partitions
    @KafkaListener(
        topics = "orders", 
        groupId = "notifications-group",
        concurrency = "3" // Spawns 3 parallel consumer threads inside this JVM instance
    )
    public void consumeOrder(ConsumerRecord<String, String> record) {
        System.out.printf("Partition: %d, Offset: %d, Message: %s%n",
            record.partition(), record.offset(), record.value());
    }
}

🟡 Middle Level

Consumer-to-Partition Ratios and Idle Consumers

Parallelism in Kafka is bounded strictly by the partition count of the subscribed topics:

Scenario Distribution Behavior & Resource Utilization
Consumers < Partitions (e.g., 2 consumers, 4 partitions) C1 $\to$ [P0, P1]
C2 $\to$ [P2, P3]		 Both consumers handle multiple partitions. High throughput per worker.
Consumers == Partitions (e.g., 4 consumers, 4 partitions) C1 $\to$ P0, C2 $\to$ P1
C3 $\to$ P2, C4 $\to$ P3		 Optimal balance. Maximum parallelism without wasted compute resources.
Consumers > Partitions (e.g., 6 consumers, 4 partitions) C1..C4 $\to$ P0..P3
C5, C6 $\to$ IDLE		 Consumers 5 and 6 remain completely idle (standby), consuming no records until an active consumer crashes.

Offset Storage Architecture: The __consumer_offsets System Topic

In early Kafka versions (pre-0.9), committed consumer offsets were stored in ZooKeeper, which created massive write bottlenecks and cluster instability. Modern Kafka stores all consumer group offsets internally in a dedicated, highly optimized system topic named __consumer_offsets:

  • Configured with 50 partitions and a replication factor of 3 by default.
  • Record Key: [group.id, topic, partition] (identifies the specific consumer stream).
  • Record Value: [offset, leaderEpoch, metadata, commitTimestamp].
  • Cleanup Policy: cleanup.policy = compact (Kafka retains only the latest committed offset record for each group-topic-partition key, discarding historical commits).

What Triggers a Consumer Group Rebalance?

A Rebalance is the distributed protocol that redistributes partition assignments among consumers in a group. It is triggered by the broker in four core events:

  1. Member Addition: A new consumer joins the group (e.g., autoscaling a microservice up).
  2. Member Departure: An existing consumer shuts down cleanly or crashes (e.g., pod killed or node hardware failure).
  3. Partition Alteration: An administrator increases the partition count of a subscribed topic.
  4. Processing Livelock: An active consumer fails to invoke poll() within max.poll.interval.ms.

🔴 Senior Level

Rebalance Protocols: Eager vs. Cooperative Sticky Rebalance

Dimension Legacy Eager Protocol (Kafka < 2.3) Cooperative Sticky Protocol (Kafka 2.3+ / KIP-429)
Processing Impact Stop-the-World: All consumers revoke all partitions and freeze processing (5–30+ seconds). Incremental: Active consumers keep processing unaffected partitions without pausing.
Partition Revocation 100% of partitions revoked globally across all members. Only the specific partitions migrating to new members are revoked.
Kubernetes Impact Severe “rebalance storms” during rolling updates, leading to cascading lags and timeouts. Smooth, transparent rolling updates; zero disruption on unmigrated partitions.
Configuration RangeAssignor / RoundRobinAssignor org.apache.kafka.clients.consumer.CooperativeStickyAssignor 
# Recommended configuration for modern mission-critical consumer groups:
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Static Membership (KIP-345): Eliminating Rebalance Storms in Kubernetes

In cloud-native container environments like Kubernetes, pods restart frequently during deployments, node drains, and canary rollouts. Under dynamic membership, every single pod restart triggers two disruptive rebalances:

  1. Pod terminates $\to$ Member leaves group $\to$ Global rebalance.
  2. New pod starts $\to$ Member joins group $\to$ Global rebalance.

The Solution — Static Membership: By assigning a persistent, unique identity to each consumer instance:

group.instance.id=payment-service-pod-0
  • When a pod restarts, the group coordinator does not trigger an immediate rebalance.
  • The coordinator holds the pod’s existing partition assignments for up to session.timeout.ms (e.g., 45 seconds).
  • When the restarted pod returns with the same group.instance.id, it instantly resumes reading its assigned partitions without triggering a group-wide rebalance.

Group Coordinator Selection Mechanics

The broker responsible for coordinating a given consumer group is selected mathematically:

  1. The client determines the partition of the offset topic:

    \[\text{offsetPartition} = \text{Utils.abs}(\text{groupId.hashCode()}) \pmod{50}\]
  2. The Kafka broker acting as the Leader replica for that specific __consumer_offsets partition automatically becomes the Group Coordinator for that group.
  3. All group members direct their JoinGroup, SyncGroup, Heartbeat, and OffsetCommit requests to this specific broker.

4 Tricky Questions

1. Why did the legacy Eager Rebalance protocol cause catastrophic “rebalance storms” during Kubernetes rolling updates, and how does Cooperative Sticky Assignor eliminate them?

Answer:

  • Under Eager Rebalancing, whenever any pod restarted, the coordinator revoked every single partition from every running instance in the group. All consumer threads halted message processing during the JoinGroup and SyncGroup phases.
  • During a rolling deployment of 20 pods, this triggered 40 consecutive Stop-the-World pauses. The accumulated consumer lag overwhelmed pod memory and CPU upon resumption, triggering extended Garbage Collection pauses that breached max.poll.interval.ms. Consumers were evicted from the group, triggering further rebalances in an endless, self-sustaining failure loop (“Rebalance Storm”).
  • Cooperative Sticky Rebalance operates incrementally. Unaffected consumers never revoke their partitions and continue processing without interruption. Only partitions migrating between pods are paused, reducing group reassignment latency from minutes to milliseconds.

2. Why can a consumer be expelled from a Consumer Group due to a timeout even though its background Heartbeat Thread is running perfectly and sending heartbeats?

Answer:

  • Since Apache Kafka 0.10.1, the Java consumer runs an internal architecture with separated threads:
    • The Heartbeat Thread sends background ping requests to the coordinator every heartbeat.interval.ms (default 3s) to prevent session.timeout.ms expiration.
    • The Application Thread runs the user’s processing logic and periodically invokes consumer.poll().
  • If the application thread hangs on a long-running database lock, slow external HTTP call, or complex computation that exceeds max.poll.interval.ms (default 5 minutes):
  • The coordinator assumes the consumer thread is deadlocked or livelocked. Despite valid heartbeats arriving from the background thread, the broker forcefully expels the consumer from the group and initiates a rebalance.
  • When the hung consumer thread finally finishes and attempts to commit offsets, Kafka throws a CommitFailedException.

3. How does Kafka select which physical broker serves as the Group Coordinator, and why does this matter for cluster load balancing?

Answer:

  • The coordinator is derived via hashing: Math.abs(groupId.hashCode()) % offsets.topic.num.partitions (default 50).
  • The broker hosting the leader replica of that computed __consumer_offsets partition becomes the Group Coordinator.
  • Load Balancing Implication: If an enterprise provisions hundreds of consumer groups that happen to share naming conventions yielding hash collisions onto the same offset partition, a single broker ends up coordinating all heartbeats, joins, and commits for the entire system.
  • Designing descriptive, differentiated group naming conventions ensures uniform distribution of coordination workloads across all cluster brokers.

4. What happens if a Kubernetes pod configured with Static Membership (group.instance.id) is killed by OOMKilled and fails to restart within session.timeout.ms?

Answer:

  • With Static Membership, the coordinator preserves partition ownership for the disconnected instance until session.timeout.ms (e.g., 45 seconds) elapses.
  • During these 45 seconds, the assigned partitions remain completely unread (“frozen”), and incoming records accumulate consumer lag.
  • If the pod fails to recover before session.timeout.ms expires, the coordinator formally registers member death, removes it from group membership, and initiates a rebalance to redistribute its partitions among the surviving live members.

🎯 Interview Cheat Sheet

Core Concepts at a Glance

  • Definition: Cooperative pool of consumers sharing partitions under a unified group.id.
  • Partition Invariant: 1 topic partition is consumed by at most 1 member per group at any instant.
  • Standby Members: Excess consumers exceeding partition count remain idle.
  • Pub/Sub Isolation: Different group.id values consume the topic independently with isolated offset trackers.
  • Offset Storage: Recorded in the internal compacted topic __consumer_offsets (50 partitions).
  • Coordinator Broker: Leader of __consumer_offsets partition derived via abs(groupId.hashCode()) % 50.
  • Cooperative Sticky Assignor: Modern non-blocking assignment protocol replacing Stop-the-World Eager rebalancing.
  • Static Membership: Assigning group.instance.id eliminates rebalances during rolling container restarts.

Rebalance Protocol Comparison

Metric Eager Protocol Cooperative Sticky Protocol
Pipeline Interruption Group-wide Stop-the-World pause Zero pause on unaffected partitions
Partition Revocation Global revocation of all partitions Incremental revocation of moving partitions
Cloud / K8s Suitability Prone to rebalance storms Cloud-native standard for containerized workloads
Default In Kafka < 2.3 Modern Kafka (2.4+)

Red Flags to Avoid

  • ❌ “Two consumers in the same group can read from the same partition concurrently to double processing speed.” (Kafka guarantees single-consumer partition exclusivity).
  • ❌ “Kafka consumer offsets are managed and stored in Apache ZooKeeper.” (Deprecated since Kafka 0.9; managed internally via __consumer_offsets).
  • ❌ “If the heartbeat thread is alive, Kafka will never evict the consumer.” (Exceeding max.poll.interval.ms will trigger forced eviction regardless of heartbeats).
  • ❌ “Enabling static membership means a crashed pod will never trigger a rebalance.” (If session.timeout.ms expires without recovery, a rebalance is triggered).