Background Jobs

Before this

• Message Queues — Hand the work to someone else and answer now — the backbone of every responsive backend.

Why this exists

When an HTTP endpoint attempts to generate a PDF, process a video, or send outbound webhooks synchronously within the request-response lifecycle, requests timeout under load, web worker pools exhaust their threads, and transient network glitches become hard user-facing 500 errors. Deferring heavy or unreliable work requires acknowledging the request immediately with an HTTP `202 Accepted` and handing the execution off to an asynchronous execution pipeline.

The mental model

A diner order spindle: The cashier writes your order on a ticket, stamps it with an order ID, hands you a receipt (HTTP `202`), and clips the ticket to the kitchen carousel. Line cooks pull tickets at their own pace. If a cook drops a plate (transient failure), the ticket gets re-pinned for a retry; if an order is missing impossible ingredients (poison pill), it is moved to an 'exception tray' (Dead-Letter Queue) so the kitchen line never grinds to a halt.

How it works

The mechanism

1. **Enqueue:** The web API pushes `{job_id, payload, attempts: 0}` to the job queue and returns HTTP `202`. 2. **Claim & Lock:** Worker claims the job; the broker starts a visibility timeout. 3. **Execute:** The worker checks its local or external storage using the `idempotency_key` to ensure work has not already been applied. 4. **Failure Path:** If the task fails with an unhandled exception or network error, the worker increments `attempts` and requeues the job with an exponential backoff delay. 5. **DLQ Routing:** If `attempts >= max_retries`, the message is pushed to the Dead-Letter Queue and an alert is emitted to the observability platform.

sequenceDiagram
  autonumber
  Client->>+API: POST /reports (Generate CSV)
  API->>Queue: Push Job (id: job_123, attempts: 0)
  API-->>-Client: 202 Accepted (Location: /jobs/job_123)
  Queue->>+Worker: Deliver Message (Visibility Lock 30s)
  Worker->>Worker: Execute Task (Fails: Downstream Timeout)
  alt attempts < MAX_RETRIES
    Worker->>Queue: Requeue with Backoff Delay (attempts: 1)
  else attempts >= MAX_RETRIES
    Worker->>DLQ: Route to Dead-Letter Queue
    Worker->>Monitoring: Emit DLQ Metric Alert
  end
  Worker-->>-Queue: ACK / Delete Original Message

What people get wrong

Queues guarantee that each job will only ever be executed exactly once.

Background queues provide at-least-once delivery; network partitions and worker crashes during execution mean duplicate deliveries are inevitable. If a worker successfully performs the action but crashes milliseconds before sending the ACK to the queue, the broker assumes failure and redelivers the message. Idempotency handling in worker logic is mandatory.

Exponential backoff alone is enough to protect failing downstream services.

Backoff without random jitter will synchronize retry waves into periodic spikes that repeatedly overwhelm the recovering service. If 1,000 tasks fail at second zero with identical backoff math (e.g., 2, 4, 8 seconds), all 1,000 workers will concurrently slam the downstream service at exactly second 2, second 4, and second 8.

In-memory process queues (like Node event loops or Go channels) are sufficient for background tasks.

In-memory queues lose all pending jobs instantly when the process restarts, deploys, or crashes under OOM. Background jobs require external durable persistence (Redis, disk-backed brokers, SQL tables) so that state survives server crashes and rolling deployments.

When not to use it

Operations requiring strict read-your-writes consistency within the immediate synchronous HTTP cycle (e.g., user password changes or authentication credential validation).

Synchronous database transactions directly inside the request handler.

High-throughput, sub-millisecond inter-thread computation where persistence overhead is prohibitive.

In-memory lock-free ring buffers (such as LMAX Disruptor) or bounded thread pools without broker hops.

Terms

Visibility Timeout

— The period during which a message broker makes an in-flight job invisible to other consumers while a worker processes it.

Dead-Letter Queue (DLQ)

— A secondary holding queue for messages that have repeatedly failed processing and exceeded the maximum retry limit.

Full Jitter

— Adding a uniform random delay between zero and the calculated exponential backoff ceiling to desynchronize retry spikes.

Poison Pill

— A malformed or invalid job payload that causes a worker to crash every time it is processed.

Idempotency Key

— A unique identifier attached to a job that allows the worker to verify whether the underlying side effect has already been applied.

Connected ideas

• Monitoring & Alerting — SLOs, error budgets, and alerts that fire on user pain rather than CPU graphs.