Message Queues
Hand the work to someone else and answer now — the backbone of every responsive backend.
50 minDifficulty 3/5distributed · architectureReviewedRead and accepted by a person.
Before this
Why this exists
A user uploads a video. Your handler transcodes it, generates thumbnails, notifies subscribers, and updates the search index — then returns. Ninety seconds later. The browser gave up at thirty. The work completed perfectly and the user saw a timeout.
The mental model
A restaurant order rail. The waiter clips the ticket to the rail and immediately goes back to the floor. Cooks take tickets when they're free. Nobody stands still waiting, and a sudden rush becomes a longer rail rather than a collapsed service.
How it works
Accept the work, then answer
A queue splits "we've got this" from "it's finished". The handler validates the request, puts a message on the queue, and responds in milliseconds. A separate worker process picks it up and does the slow part. The user gets an immediate answer, and if the honest answer is "not done yet", that's what `202 Accepted` is for.
It absorbs bursts instead of collapsing under them
Without a queue, a spike to ten times normal traffic means ten times the concurrent work, and something falls over — connection pools exhaust, memory runs out, everything degrades at once. With a queue, a spike makes the queue longer. Workers keep consuming at their steady rate, latency rises gracefully, and nothing breaks. The queue is a shock absorber: it converts a capacity problem into a delay problem, which is nearly always the better failure.
At-least-once delivery, so consumers must be idempotent
A worker takes a message, does the work, and acknowledges it. If it crashes before acknowledging, the queue can't tell whether the work happened — so it redelivers. That means every consumer will eventually process the same message twice, and this is by design. **Idempotency is not optional here**; it's the price of not losing messages. The alternative, acknowledging before working, loses the message whenever a worker dies mid-job.
Failures need somewhere to go
A message that always fails — malformed data, a deleted record — will be retried forever, consuming capacity and burying healthy messages. That's a **poison message**. The standard defence is a retry limit with exponential backoff, after which the message moves to a **dead letter queue**: a separate queue nobody consumes automatically, where failures accumulate for a human to inspect. A DLQ with items in it is one of the highest-signal alerts you can have — it means something is broken *and* you didn't lose the evidence.
The mechanism
The delivery cycle, and where each guarantee comes from: 1. Producer enqueues. The message is durable — it survives a broker restart. 2. A worker receives it. The message becomes **invisible** to other workers for a visibility timeout, rather than being deleted. 3. The worker does the job. 4. On success it acknowledges, and the message is deleted for good. 5. On failure — or if the worker dies and the visibility timeout expires — the message becomes visible again and is redelivered. 6. After N attempts it goes to the dead letter queue. Step 2 is the subtle one. The message isn't removed on receipt, only hidden. That's exactly what makes crash recovery automatic — and exactly what makes duplicates inevitable.
flowchart LR
A[API handler] -->|enqueue| B[(Queue)]
A -->|202 immediately| C[Client]
B -->|deliver| D[Worker 1]
B -->|deliver| E[Worker 2]
D -->|ack| B
E -->|nack after N tries| F[(Dead letter queue)]
F --> G[Human investigates]What people get wrong
- A queue makes the work faster.
- The work takes just as long. What changes is that the user stops waiting for it. You're moving work off the request path, not speeding it up. Throughput improves only if you also add workers.
- Messages are delivered exactly once.
- At-least-once is the practical guarantee. Duplicates will happen. The broker can't distinguish a crashed worker from a slow one, so it must redeliver. True exactly-once requires cooperation from the consumer's storage.
- Queues preserve global ordering.
- With multiple workers, order is not preserved. Ordering is opt-in and costs parallelism. Two workers pulling concurrently finish in whatever order the work takes. FIFO queues and partition keys restore ordering per key, at the cost of throughput.
- Acknowledge on receipt to keep the queue moving.
- Acknowledge after the work succeeds. Acknowledging first turns at-least-once into at-most-once — a worker crash silently loses the job.
When not to use it
- The caller needs the result to continue.
- A synchronous call. A queue is for work the caller doesn't need to wait for; forcing a request-response through a queue adds latency and complexity for nothing.
- Many consumers each need every event, and need to replay history.
- An event log like Kafka. A classic queue deletes a message once it's consumed.
- You have one small app and modest background work.
- A database-backed job table with a polling worker. It's less to operate and gets you the same decoupling until you actually need more.
Terms
- Producer
- — The code that puts a message on the queue.
- Consumer / worker
- — A process that takes messages off and does the work.
- Acknowledgement
- — The signal that a message was handled successfully and may be deleted.
- Visibility timeout
- — How long a received message stays hidden from other workers before being redelivered.
- Dead letter queue
- — Where messages go after exhausting their retries, so failures are preserved rather than lost or looped.
- Backpressure
- — Signalling upstream to slow down when the queue grows faster than it drains.
- Poison message
- — A message that fails every time and would otherwise be retried forever.
In an interview
How would you handle a slow operation in an API request?
- validate synchronously, enqueue, return 202 with a way to check status
- a worker processes it separately
- the consumer must be idempotent because delivery is at-least-once
- retries with backoff and a dead letter queue for permanent failures
What happens if a worker crashes halfway through a job?
- the message wasn't acknowledged, so the visibility timeout expires and it's redelivered
- another worker picks it up and may repeat partially-completed work
- this is why consumers must be idempotent
- acknowledging before the work would lose the job instead
Can you recall it?
Why does using a message queue force every consumer to be idempotent?
Connected ideas
- Background Jobs — Retries, backoff, dead-letter queues — what happens after you say 202 Accepted.
- Idempotency — Making 'do it again' safe — the single idea that separates reliable distributed code from hope.
- Webhooks — Inverting the call — they tell you when something happened instead of you asking forever.