---

🟢 Junior Level

What is a message key?

Message Key is a value that determines which partition a message will land in. The key is the “address” for routing a message.

Why: a way to tell Kafka: “all events related to this user/order should be processed in one place.” Without a key, events for the same user would scatter across different partitions and might arrive out of order.

Analogy

Think of an airport:

• Key = flight destination code (e.g., “KYIV-BA101”)
• Partition = specific gate
• All passengers with the same code go to the same gate
• Passengers with different codes — to different gates

key="KYIV-BA101" → gate 5 (partition 1)
key="NYC-AA202"  → gate 3 (partition 0)
key="PARIS-AF303" → gate 7 (partition 2)

A better analogy: a key is a case number in court. All documents for the same case go into one folder and are processed in strict order. Documents for different cases — in different folders, order between them doesn’t matter.

Why is a key needed?

1. Ordering guarantee — messages with the same key go to one partition → preserve order
2. Grouping — all events for one user/object are processed together

Example

// With key — ordering guaranteed
ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", "user-123", "Order created");
//                    topic         key          value

// user-123 always lands in the same partition

With key:
  user-123 → partition 1 → order: login → browse → purchase

Without key:
  msg-1 → partition 2
  msg-2 → partition 0  // ordering not guaranteed
  msg-3 → partition 1

When NOT to use a message key

1. Message order doesn’t matter and maximum throughput is needed — key=null
2. Keys have strong skew (uneven distribution) — hot partition
3. Key size is too large (>1KB) — CPU overhead on hashing

---

🟡 Middle Level

Ordering guarantees

Partition 0: user-456 events (strictly in order)
Partition 1: user-123 events (strictly in order)
Partition 2: user-789 events (strictly in order)

Important: Ordering is guaranteed only within a single partition! Between partitions — no guarantees.

Hashing algorithm

partition = Math.abs(Utils.murmur2(keyBytes)) % numPartitions

Kafka uses MurmurHash 2 for uniform distribution of keys across partitions.

Choosing a good key

Key	Distribution	When to use
userId	✅ Uniform (many users)	User events
orderId	✅ Uniform	Order events
"all"	❌ Hot partition	Do not use for event-driven systems. Acceptable for broadcast (global config), but be aware of hot partition risk.
null	✅ Sticky partitioning	When ordering is not important
aggregateId	✅ Uniform	Event sourcing

Null Key

// key = null → sticky partitioning
// Ordering is NOT guaranteed
ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", "Order created");

Key strategy comparison

Strategy	Ordering	Throughput	Distribution
Business Key (userId)	✅ Within partition	Medium	Depends on data
Composite Key (userId_orderId)	✅	Medium	Good
Salted Key (userId + random)	⚠️ Partial	Good	Excellent
Null Key	❌	Maximum	Uniform

Common mistakes

Mistake	Consequence	Solution
Null key when ordering is needed	User events are mixed up	Use userId as key
Same key for everything	Hot partition	Different keys for different entities
Large key (JSON)	CPU overhead on hashing	Compact keys (UUID, Long)
Changing partition count	Key distribution changes	Create a new topic

---

🔴 Senior Level

Internal Implementation — MurmurHash 2

// org.apache.kafka.common.utils.Utils
public static int murmur2(byte[] data) {
    int length = data.length;
    int seed = 0x9747b28c;  // Kafka seed
    // ... MurmurHash2 algorithm ...
    return fmix(hash(length, seed, data));
}

// Partition calculation
int hash = Utils.murmur2(keyBytes);
int partition = Math.abs(hash) % numPartitions;

Why MurmurHash 2:

• Non-cryptographic, fast (bitwise operations)
• Avalanche effect — changing 1 bit → ~50% of result bits change
• Uniform distribution — minimum collisions for real-world data

Integer.MIN_VALUE edge case:

// Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE (negative!)
// Kafka uses a bitmask:
int partition = (hash & 0x7fffffff) % numPartitions;

Key Ordering Guarantees — formal model

Within a partition: strict FIFO (total order)
Between partitions: partial order (causal ordering only with explicit design)

Formally:
  ∀ messages m1, m2:
    key(m1) == key(m2) → partition(m1) == partition(m2) → order(m1, m2) preserved
    key(m1) != key(m2) → order(m1, m2) NOT guaranteed

Advanced Key Strategies

1. Composite Key for fighting hot partitions:

// Composite key: entity + sub-entity
String key = userId + "_" + orderId;
// Provides more even distribution
// But: ordering only within userId_orderId, not within userId

2. Key Salting (randomized partitioning within entity):

// N-ary salting: distribute events of one user across N partitions
int salt = ThreadLocalRandom.current().nextInt(10);
String key = userId + "_" + salt;
// Ordering is lost, but throughput grows N times
// Use when ordering is NOT critical

3. Time-based Key Rotation:

// Key includes a time bucket
String key = userId + "_" + (System.currentTimeMillis() / 3600000);
// Key changes every hour → messages land in new partition
// Useful for time-windowed aggregations

4. Consistent Hashing for dynamic partitions:

Ring-based approach: minimizes key movement when N changes
Libraries: ketama, jmpmv/consistent-hashing
Application: multi-tenant systems, sharded databases

Key Evolution Problem — formal analysis

Before: partition = hash(key) % 3
  key="user-1" → hash("user-1") % 3 = 1 → Partition 1

After: partition = hash(key) % 6
  key="user-1" → hash("user-1") % 6 = 4 → Partition 4

Same key → different partition → ordering broken!

Mathematically:
  P(new_partition != old_partition) = 1 - (old_N / new_N)
  For 3 → 6: P = 1 - 3/6 = 50% of keys move
  For 10 → 100: P = 1 - 10/100 = 90% of keys move

Solutions:

1. Plan partition count with margin (10x current need)
2. Use consistent hashing (custom partitioner)
3. Create a new topic with the desired partition count and migrate via dual-write

Key Size Impact on Performance

Key size	Memory overhead	CPU (hash)	Network	Recommendation
Long (8 bytes)	Minimal	~5 ns	Minimal	✅ Optimal
UUID (36 bytes)	Medium	~20 ns	Medium	✅ Acceptable
String (100 bytes)	Noticeable	~50 ns	Noticeable	⚠️ Acceptable
JSON (10 KB)	Significant	~500 µs	Significant	❌ Avoid

Memory footprint:

Key is stored in:
- Producer RecordBatch → MemoryRecord (heap)
- Broker Log → MessageSet (page cache + heap)
- Consumer ConsumerRecord (heap)

Total: 3 × key_size × throughput × retention
For 10KB keys at 10K msg/s = ~3 GB/s memory traffic

Edge Cases (3+)

1. Key Null + Idempotent Producer: With enable.idempotence=true and key=null, Kafka generates sequence numbers at the partition level. Since sticky partitioning switches partitions, sequence numbers don’t conflict. But if partitioner.class is custom and non-deterministic — OutOfOrderSequenceException may occur.

2. Key Serialization Schema Mismatch: Producer serializes key as String (StringSerializer), consumer deserializes as ByteArray (ByteArrayDeserializer). Key “works” for partitioning, but consumer business logic receives byte[] instead of the expected String. Test serialization on both ends.

3. Unicode Keys and MurmurHash: MurmurHash operates on bytes. Key "café" in UTF-8 = [99, 97, 102, 195, 169], in Latin-1 = [99, 97, 102, 233]. Different encodings → different bytes → different partitions. Ensure all producers use the same encoding (UTF-8 by default in Kafka).

4. Key Mutation in Processor: In Kafka Streams or ksqlDB, if a processor changes the message key (map((key, value) -> new KeyValue<>(newKey, value))), the message is redistributed to a new partition. This causes a shuffle — an expensive operation. Use through() for explicit repartitioning control.

5. Empty Key (not null, but empty string): key="" → MurmurHash("") = constant → all messages with empty key go to one partition. This is like a hot partition, but harder to detect (key “exists” but is useless).

Performance Numbers

Metric	Value	Conditions
MurmurHash latency	~5-50 ns	Depends on key size
Partition compute overhead	< 100 ns	Including serialization
Key serialization (String, 36 bytes)	~100 ns	UTF-8 encoding
Impact of hot key on p99	3-10x latency increase	1 key = 50% of traffic

Production War Story

Situation: Ride-sharing company used driverId as key for topic rides. With 10,000 active drivers, distribution was even (20 partitions). But during peak hours, the top 50 drivers (most active) generated 40% of all events.

Problem: P99 latency for orders grew from 15ms to 500ms. Lag on “hot” partitions reached 30K messages. Clients complained about “stuck” orders.

Analysis:

Top-10 keys by volume:
  driver-42: 15K msg/min → Partition 7
  driver-17: 12K msg/min → Partition 3
  driver-88: 11K msg/min → Partition 14
  Average: 50 msg/min per driver

Solution:

1. Introduced composite key: driverId + "_" + (timestamp / 60000) — key changes every minute
2. Increased partitions from 20 to 60
3. For aggregations (count rides per driver) used Kafka Streams with .groupByKey() — automatic repartition
4. Set up alert on per-partition throughput imbalance > 3x

Lesson: Uniform key distribution ≠ uniform load. Analyze frequency distribution of keys, not just cardinality.

Monitoring (JMX + Burrow)

JMX metrics:

kafka.producer:type=ProducerMetrics,client-id=producer-1,key=record-send-rate
kafka.producer:type=ProducerMetrics,client-id=producer-1,key=batch-size-avg
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

Custom monitoring:

// Interceptor for key distribution analysis
public class KeyDistributionInterceptor implements ProducerInterceptor {
    private final Map<String, Long> keyCounts = new ConcurrentHashMap<>();

    public ProducerRecord onSend(ProducerRecord record) {
        keyCounts.merge(String.valueOf(record.key()), 1L, Long::sum);
        return record;
    }
}

Highload Best Practices

1. Key = compact business entity ID (Long > UUID > String)
2. Analyze frequency distribution of keys before production
3. Composite/salted keys for fighting hot partitions (trade-off: ordering)
4. Avoid changing partition count — create a new topic
5. Test on production-like data — real keys have skew
6. Monitor per-partition throughput — alert on imbalance > 3x
7. Empty string != null — empty key means hot partition
8. Unicode consistency — all producers must use UTF-8

---

🎯 Interview Cheat Sheet

Must know:

• Key determines partition: MurmurHash2(key) % numPartitions
• Messages with the same key always land in one partition → ordering guaranteed
• Without key (null) — Sticky Partitioning, no ordering
• Composite key (userId + "_" + orderId) — for even distribution
• Large key (>1KB) — CPU overhead on hashing every message
• When changing partition count, same key → different partition
• Empty string ≠ null — empty key = hot partition (all to one partition)
• Key = compact business entity ID (Long > UUID > String)

Common follow-up questions:

• What happens if key=null? — Sticky Partitioning, ordering not guaranteed.
• Why not use JSON as a key? — 10KB key × 10K msg/s = ~3 GB/s memory traffic.
• How to ensure ordering with hot key? — Composite/salted key, trade-off: partial loss of ordering.
• What is the Key Evolution Problem? — When changing N in hash(key) % N, 50-90% of keys move.

Red flags (DO NOT say):

• “Key guarantees global ordering” — only within one partition
• “Empty string is the same as null” — empty = hot partition, null = sticky
• “You can change partition count without consequences” — keys move
• “Key is stored only in the producer” — stored in RecordBatch, Log, ConsumerRecord (3×)

Related topics:

• [[3. How is data distributed across partitions]]
• [[2. What is partition and why is it needed]]
• [[21. What is batch in Kafka producer]]
• [[1. What is topic in Kafka]]