System design deep dive
Message Queue: deep dive
A message queue lets one service hand off work to another without waiting for it, so spikes are absorbed, slow tasks move off the request path, and a failing consumer does not take the producer down with it. Interviews focus on delivery guarantees, ordering, and what happens when things go wrong.
How a message flows through a queue, step by step
- A user uploads a video. The API stores the raw file in object storage and publishes a small message
{videoId: 7731, action: transcode}to the queue, then immediately returns 202 Accepted. - The broker persists the message to disk (and to replicas, in Kafka or a replicated RabbitMQ quorum queue) before acknowledging the publish to the producer.
- A transcoding worker pulls the message. In SQS it becomes invisible to other consumers for a visibility timeout, for example 5 minutes; in Kafka the consumer simply reads from its assigned partition at its current offset.
- The worker processes the job and then acknowledges it (SQS delete, RabbitMQ ack, or a Kafka offset commit).
- If the worker crashes before acknowledging, the message becomes visible again or the offset is re-read, and another worker retries it. This is why delivery is at-least-once.
- After a configured number of failed attempts, say 5, the message moves to a dead-letter queue for inspection instead of blocking the pipeline forever.
Queue vs log: RabbitMQ/SQS vs Kafka
| Aspect | Traditional queue (SQS, RabbitMQ) | Distributed log (Kafka, Pulsar) |
|---|---|---|
| Message lifetime | Deleted once acknowledged | Retained for days regardless of consumption |
| Multiple consumers | Competing consumers share the work | Each consumer group reads the full stream independently |
| Replay | Not possible after ack | Rewind the offset and reprocess history |
| Ordering | Best effort (FIFO queues optional) | Strict within a partition |
| Throughput | Thousands to tens of thousands msg/s per queue | Millions msg/s across a cluster |
| Typical use | Background jobs, task distribution | Event streaming, CDC, analytics pipelines |
When to use a queue (and when not to)
Use a queue when the caller does not need the result right away: sending emails and push notifications, generating thumbnails, updating search indexes, fan-out of events to many services, or smoothing a traffic burst so a database receives a steady 2,000 writes per second instead of a 20,000 per second spike. Avoid a queue when the user is waiting on the answer (checking a password, reading a balance); a synchronous call is simpler and a queue only adds latency and failure modes. Also avoid using a queue as a database: it is a transport, and long-term state belongs in a store you can query.
Making consumers idempotent
Because redelivery is normal, every consumer should produce the same result if it sees the same message twice. The usual technique is a deduplication record keyed by a message ID, written in the same transaction as the side effect.
BEGIN;
-- Fails with a unique violation if this message was already handled.
INSERT INTO processed_messages (message_id, processed_at)
VALUES ('evt_5f2a91', now());
UPDATE wallets SET balance = balance + 500 WHERE user_id = 42;
COMMIT;
-- On unique violation: roll back and ack the message; it is a duplicate.Common mistakes in interviews
- Claiming exactly-once delivery without explaining how. Real systems get effectively-once by combining at-least-once delivery with idempotent consumers or transactional writes.
- Writing to the database and then publishing to the queue as two separate steps. If the process dies in between, the event is lost. Use the transactional outbox pattern.
- Requiring global ordering, which forces a single partition and kills throughput. Usually ordering per entity (per order ID, per user) is enough, achieved by using that ID as the partition key.
- No dead-letter queue, so one poison message blocks or endlessly retries.
- Not monitoring consumer lag, the clearest signal that consumers are falling behind.
Interview questions
›What is the transactional outbox pattern?
Instead of publishing directly, the service writes the event into an outbox table in the same database transaction as the business change. A separate relay process (or change data capture tool such as Debezium) reads the outbox and publishes to the broker. The event is published if and only if the transaction committed.
›How does Kafka scale consumers?
A topic is split into partitions, and within a consumer group each partition is assigned to exactly one consumer. Throughput scales by adding partitions and consumers, but you cannot usefully have more consumers in a group than partitions, so partition count is chosen with future scale in mind.
›What is backpressure and how do queues provide it?
Backpressure is a way for a slow downstream to avoid being overwhelmed by a fast upstream. A queue buffers the excess so consumers process at their own pace, and the growing queue depth signals that you should add consumers. If the buffer can fill up, you also need a policy: block producers, drop low-priority messages, or reject with an error.
›How do you retry failed messages without hammering a broken dependency?
Use exponential backoff with jitter between attempts, for example 1s, 4s, 16s, then park the message in a dead-letter queue. Some systems use separate delay queues per retry tier. Pair this with a circuit breaker so consumers pause when the dependency is clearly down.
›When would you choose pub/sub over a point-to-point queue?
Choose pub/sub when several independent services need to react to the same event, such as an OrderPlaced event consumed by billing, shipping, and analytics. Point-to-point fits when a message represents one unit of work that exactly one worker should perform.