System Design · Hard
Design a distributed message queue
Build a Kafka-like log: partitioned, replicated, with consumer groups and exactly-once semantics.
Requirements
What the system must do, and how well it must do it.
Functional
Producers append messages to topics (partitioned by key).
Consumers read in consumer groups (each partition → one consumer).
Message retention: time-based (7 days) or size-based (1 TB).
Exactly-once semantics: idempotent producers + transactional consumers.
Replication factor 3; ISR (in-sync replicas) for durability.
Non-functional
p99 produce latency under 10 ms.
Throughput: 10 MB/sec/partition, 1M msg/sec/broker.
Zero data loss: ack=all waits for ISR.
Consumer lag monitoring and alerting.
Estimation
Back of the envelope numbers that drive the design.
100 topics, 100 partitions each = 10K partitions.
3 brokers × 10 TB = 30 TB raw. 10K partitions × 3 replicas = 30K partition replicas.
Retention: 7 days × 100 GB/day = 700 GB per topic average.
Consumer group rebalance: < 30 sec for 1K consumers.
Architecture
Design decisions
Partitioned log
A topic is split into partitions. Each partition is an append-only log of messages with offsets. The partition key (hash of message key) determines partition assignment. Ordering is guaranteed per partition. Partitions are distributed across brokers for parallelism.
Replication and ISR
Each partition has N replicas (default 3). One is the leader; followers replicate from the leader. The ISR (in-sync replicas) are followers caught up to the leader. acks=all means the leader waits for all ISR replicas to acknowledge before responding to the producer. If a follower falls behind, it is removed from ISR; if the leader fails, a new leader is elected from ISR.
Consumer groups
Consumers form a group. Each partition is assigned to exactly one consumer in the group. The group coordinator (a broker) manages assignment via a protocol (range, round-robin, sticky). On rebalance (consumer join/leave), partitions are reassigned. Consumers commit offsets to a special __consumer_offsets topic (compacted).
Exactly-once semantics
Idempotent producer: each message gets a PID + sequence number per partition. The broker deduplicates on receipt. Transactional producer: beginTransaction, send (multiple partitions), commitTransaction — all or nothing. Transactional consumer: read-process-write in a single transaction (consume-transform-produce). The offset commit is part of the transaction.
Questions and answers
Follow-ups an interviewer is likely to ask. Try answering before you open each one.
How do you handle a slow consumer?
Monitor consumer lag (high-water-mark - committed-offset). Alert if lag > threshold. Scale the consumer group (add consumers) or increase partition count (requires topic re-partitioning). Dead letter queue for poison pills.
What if a broker loses its disk?
The broker restarts, fetches missing data from ISR replicas (if any), or replicates from the new leader. If no ISR replicas (catastrophic), the partition is under-replicated until the broker catches up. Data loss only if all replicas lost simultaneously.
How do you implement delayed/retry queues?
Use a separate retry topic with a TTL header. A delayed consumer reads from the main topic, writes to the retry topic with a timestamp. A timer service (or Kafka Streams) moves messages back to the main topic when the delay expires. Or use a plugin like Kafka Delayed Operations.