Junior Level

Short answer

Technically — yes, you can, but the extra consumers won’t do any work.

The “1 partition — 1 consumer” rule

Within one Consumer Group:
A single partition can only be read by one consumer

Example

5 partitions, 8 consumers:
  Consumer 1 → Partition 0 (working)
  Consumer 2 → Partition 1 (working)
  Consumer 3 → Partition 2 (working)
  Consumer 4 → Partition 3 (working)
  Consumer 5 → Partition 4 (working)
  Consumer 6 → idle
  Consumer 7 → idle
  Consumer 8 → idle

Why is it this way?

Kafka guarantees strict ordering of message processing within a partition. If two consumers read the same partition, there would be offset contention and ordering would break.

---

Middle Level

What happens to “extra” consumers?

Extra consumers:
- Maintain connection to broker (heartbeat)
- Don't receive data for processing
- Consume resources (RAM, CPU, network)
- Ready to pick up partitions on rebalance

When can this be useful? (Hot Standby)

Scenario: high availability

5 partitions, 7 consumers:
  5 active → process data
  2 standby → ready to take over on crash

If Consumer 1 crashes:
  Rebalance → Consumer 6 picks up Partition 0
  Recovery time is minimal

How to actually increase parallelism?

1. Increase partition count:

kafka-topics.sh --alter --topic orders --partitions 10
# Now you can run 10 consumers

2. Optimize processing code:

// Async processing (be careful with ordering)
for (var record : records) {
    asyncProcess(record);  // doesn't block poll
}

3. Thread Pool within a consumer:

// ⚠️ Breaks message ordering!
ExecutorService executor = Executors.newFixedThreadPool(10);
for (var record : records) {
    executor.submit(() -> process(record));
}
consumer.commitSync();  // commit after all tasks

Exception: Different Consumer Groups

One topic "orders" (5 partitions) read by 10 groups:

Group 1: 5 consumers → all partitions
Group 2: 5 consumers → all partitions
...
Group 10: 5 consumers → all partitions

Total: 50 consumers reading one topic
Each group gets a full copy of the data!

Common mistakes

1. Launching extra consumers for no reason:

   Resources wasted (RAM, CPU, connections)
   

2. Expecting increased throughput:

   More consumers ≠ more throughput
   Throughput is limited by partition count
   

3. Not understanding group.id:

   Different applications with the same group.id
   → Share partitions → each gets only partial data
   

---

Senior Level

Internal Implementation

Group Coordinator stores:

Member list → list of all consumers in the group
Partition assignments → which partition goes to which consumer
Committed offsets → last committed offset

Idle consumers:
- In the member list
- Have no assigned partitions
- Send heartbeats
- Participate in rebalance

Resource Consumption

Each consumer consumes:
- RAM ~100-200MB (JVM)
- CPU for heartbeat processing
- Network connection to broker
- File descriptor for socket
- Entry in __consumer_offsets

10 idle consumers = wasted resources

Scaling Strategies

1. Partition Count Planning:

Formula: partitions = max(producer_throughput, consumer_throughput)

Example:
  Need: 100 MB/s
  One consumer: 10 MB/s
  Minimum partitions: 10
  Recommendation: 12-15 (growth margin)

2. Async Processing within consumer:

// Preserve ordering for a single key
public class OrderPreservingAsyncProcessor {
    private final Map<String, CompletableFuture<Void>> pending = new HashMap<>();

    public void process(ConsumerRecord<String, String> record) {
        String key = record.key();
        pending.compute(key, (k, future) -> {
            if (future == null) {
                return processAsync(record);
            }
            return future.thenRun(() -> processAsync(record));
        });
    }
}

3. Batch Processing Optimization:

// Increase max.poll.records for throughput
props.put("max.poll.records", "1000");
// More messages per poll → more throughput

Hot Standby Pattern

# High availability configuration
group.id: order-processors
group.instance.id: consumer-${HOSTNAME}
session.timeout.ms: 10000        # fast detection
heartbeat.interval.ms: 3000
max.poll.interval.ms: 300000

# Launch N+2 consumers for N partitions
# 2 extra — hot standby

Advantages:

• Recovery within session.timeout.ms (typically 10-45 seconds). Not instant!
• Minimal downtime
• Automatic failover

Disadvantages:

• Resource consumption without benefit
• Monitoring complexity (who is active?)
• Extra connections to broker

When NOT to use Hot Standby

Hot Standby is not worth using when: resources are limited, load is stable and predictable, SLA allows 30-60 second recovery.

Cross-Group Coordination

Scenario: different business tasks

Topic "orders" (10 partitions):
  Group "payment-processing" → 10 consumers
  Group "analytics" → 5 consumers
  Group "notification" → 3 consumers
  Group "audit" → 10 consumers

Total: 28 consumers, 10 partitions
Each group is independent

Performance Analysis

Consumer Lag Analysis:
  Lag growing → consumers can't keep up

Options:
  1. Increase partitions (and consumers)
  2. Optimize per-message processing
  3. Increase max.poll.records
  4. Async processing (with ordering loss)

Best Practices

✅ Consumer Count == Partition Count — ideal balance for throughput. For high availability use N+1 or N+2 (hot standby).
✅ Hot Standby for critical systems (N+1 or N+2)
✅ Increase partitions for scaling
✅ Async processing when ordering is not critical
✅ Different Consumer Groups for different business tasks

❌ More consumers than partitions without reason
❌ Expecting increased throughput from extra consumers
❌ Same group.id for different applications
❌ Thread pool without ordering control

Architectural decisions

1. Plan partitions ahead — they determine maximum parallelism
2. Hot Standby justified for HA — but requires monitoring
3. Async processing — a trade-off — throughput vs ordering
4. Different groups for different tasks — independent scaling

Summary for Senior

• Extra consumers = hot standby, consuming resources
• Balance Consumer Count == Partition Count — ideal
• For scaling, start with partition planning
• Async processing increases throughput at the cost of ordering
• Different Consumer Groups enable independent reading

---

🎯 Interview Cheat Sheet

Must know:

• Technically possible, but extra consumers are idle — receive no data
• Rule: 1 partition = maximum 1 active consumer in the group
• Extra consumers = hot standby: consume RAM/CPU, ready to take over on failover
• Throughput is limited by partition count, not consumer count
• Real scaling = more partitions + more consumers
• Hot Standby justified for HA (N+1 or N+2), but requires monitoring
• Different Consumer Groups = independent reading of one topic
• Thread pool within a consumer breaks message ordering

Common follow-up questions:

• Why launch extra consumers? — Hot standby for fast failover.
• How to increase parallelism without adding partitions? — Async processing (trade-off: ordering), thread pool.
• Can you have 50 consumers on 5 partitions? — Yes, but 45 will be idle.
• How do different groups read one topic? — Each group with unique group.id gets a full data copy.

Red flags (DO NOT say):

• “More consumers = more throughput” — throughput is limited by partitions
• “Idle consumers are free” — they consume RAM, CPU, network connections
• “Thread pool preserves ordering” — breaks ordering within a partition
• “Groups share data between themselves” — each group gets a full copy

Related topics:

• [[5. What is Consumer Group]]
• [[6. How does consumer balancing work in a group]]
• [[8. What happens when you add a new consumer to the group]]
• [[2. What is partition and why is it needed]]