Redis Queues

What is a job queue?

A job queue is a durable buffer that sits between the code that asks for work and the code that performs it. The producer writes a small record describing a unit of work and returns immediately. A worker reads that record later and does the slow part. The queue is what survives a restart, absorbs a burst, and gives a failure somewhere to go.

Redis has been used this way for more than a decade. Its lists gave early queue libraries an atomic LPUSH and BRPOP primitive, and blocking pops let a worker sleep until work arrived instead of polling in a loop. Streams later added consumer groups, acknowledgements and replay. On top of those primitives, BullMQ wraps the whole lifecycle — delayed jobs, retries, priorities, rate limiting, repeatable schedules and events — in a small TypeScript API.

If Redis already backs your cache or your sessions, adding a queue is a short step. That convenience is the main reason it is the default choice for Node teams, and the main reason to understand both what it gives you and what it does not.

A queue is three roles, not one

Every queue system, no matter the broker, has the same three roles, and keeping them distinct is what makes the system maintainable.

The producer is any code that adds a job. It knows the job name and the payload shape and nothing else. It must be fast, because it usually runs inside a request, and it should be safe to call twice.

app.post("/signup", async (req, res) => {
  const user = await db.user.create({ data: req.body });

  await emails.add("welcome", { userId: user.id, to: user.email });

  res.status(201).json({ id: user.id });
});

The queue is the shared state in the middle. In a Redis deployment it is the set of keys BullMQ maintains: the waiting list, the delayed set, the active set and the failed set. It persists jobs across restarts, hands them out atomically, and tracks attempts.

The worker is a long-running process that pulls jobs and runs handlers. It is separate from the API for good reasons: it can be deployed on CPU-optimised hardware, scaled to the queue depth, and restarted without dropping requests. One process can host workers for several queues, and one queue can be served by many worker processes. The queue is the only shared state, which is why it scales horizontally so cleanly.

Why Redis is a common backing store

Three Redis properties make it a natural queue.

First, the primitives already exist. A list with LPUSH and BRPOP is a queue, and BRPOP blocks the connection rather than burning CPU. That is the entire reason the first Node queue libraries were a few hundred lines.

LPUSH queue:emails "welcome:42"
BRPOP queue:emails 30

Second, commands are atomic. Moving a job from waiting to active, incrementing its attempt count and setting its lock can all happen without a race, because Redis executes commands one at a time. That is exactly what a queue needs to avoid two workers claiming the same job.

Third, Redis is probably already there. It is the default cache and session store for Node services. Reusing it for a queue avoids a second piece of infrastructure, a second set of credentials and a second runbook.

Streams extend the idea further. Where a list can only be pushed and popped, a stream is an append-only log with consumer groups, per-message acknowledgements and a replayable history.

XADD jobs:emails '*' type welcome userId 42
XREADGROUP GROUP workers alice COUNT 10 STREAMS jobs:emails '>'

The catch is durability. Redis is an in-memory store first, so a job acknowledged but not yet written to disk can be lost if the instance dies. You can narrow that window with AOF and a replica, but you cannot make Redis as durable as a write-ahead-logged database. Treat a job as recoverable work, not as your system of record.

Getting started with BullMQ

BullMQ needs a Redis connection that is allowed to retry forever. The default ioredis behavior gives up on a command after a few failures, which is wrong for a worker that must ride out a blip, so set maxRetriesPerRequest: null.

import { Queue, Worker } from "bullmq";
import IORedis from "ioredis";

const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: null,
});

const emails = new Queue("emails", { connection });

const worker = new Worker(
  "emails",
  async (job) => {
    await sendEmail(job.data.to, job.data.template);
  },
  { connection, concurrency: 10 },
);

The Queue is the producer handle. worker is the consumer. The name "emails" is the queue, and a worker only ever sees jobs added to its own queue. Separate queues are the unit of isolation: a slow import queue cannot delay a password-reset queue.

A Queue is lightweight and can be created wherever you need to add a job. A Worker is a long-running process and should be started once per process, not per request. Share the connection object across all queues and workers in the process.

Adding jobs with data and options

add takes a name, a payload and an options object. The name routes the job to a handler; the payload carries everything the worker needs.

await emails.add("welcome", { userId, to }, {
  jobId: `welcome:${userId}`,
});

The payload should be a snapshot of intent, not a live object. If a user changes their email between enqueue and execution, the job should still send to the address it was created with. Store ids and the few values that define the work rather than the whole database row, because the queue holds every waiting job in memory.

The options object is where most of BullMQ’s value lives:

• attempts — the maximum number of tries before the job is considered failed.
• backoff — the delay strategy between attempts, such as exponential.
• delay — do not run before this many milliseconds from now.
• priority — a lower number runs first when jobs are waiting.
• jobId — a stable id that deduplicates enqueues of the same logical work.
• removeOnComplete — how many completed jobs to keep, or true to remove immediately.
• removeOnFail — whether to keep failures for inspection. Keep them.

Processing jobs and returning results

A handler receives the job, does the work, and may return a value. The return value is stored on the job and can be read later, which turns the queue into a simple asynchronous RPC.

const worker = new Worker(
  "reports",
  async (job) => {
    const pdf = await renderPdf(job.data.reportId);
    return { url: pdf.url, bytes: pdf.bytes };
  },
  { connection },
);

A caller can then poll the job for its result, or listen on the events stream for completion.

const job = await reports.getJob(jobId);

if (await job?.isCompleted()) {
  return job.returnvalue;
}

if (await job?.isFailed()) {
  throw new Error(job.failedReason);
}

Return small values. A result is stored in Redis like any other data, so returning a megabyte PDF bloats the queue. Return a URL or an id and let the caller fetch the bytes from object storage.

Retries, exponential backoff and the failed set

Transient failures are normal. A database fails over, an API rate-limits you, a container is rescheduled. Retrying is the correct response, but retrying immediately is not.

await queue.add("sync", { accountId }, {
  attempts: 5,
  backoff: { type: "exponential", delay: 2_000 },
});

That produces delays of roughly 2s, 4s, 8s and 16s, with BullMQ’s own jitter applied so that many jobs which failed together do not retry together. Without jitter, a fleet of workers that all hit the same outage would recreate the spike the moment it recovered.

When a job exhausts its attempts, BullMQ does not delete it. It moves it to the failed set, along with the error message and stack trace. That set is the operational surface of the queue: alert when it grows, and build a replay path.

const failed = await queue.getFailed(0, 20);

for (const job of failed) {
  // Fix the underlying cause first, then requeue.
  await job.retry();
}

A job that fails every time is a poison message. Retrying it forever is worse than not retrying at all, because it consumes a worker slot on every attempt and starves healthy work. Cap attempts, and inspect the failed set rather than ignoring it.

Delayed and repeatable jobs

A delayed job is scheduled for later without a separate scheduler. A repeatable job runs on a cron pattern and is owned by the queue.

await queue.add("reminder", { userId }, { delay: 600_000 });

await queue.add(
  "digest",
  { region: "eu" },
  {
    repeat: { pattern: "0 7 * * *", tz: "Europe/Berlin" },
    jobId: "digest:eu",
  },
);

The stable jobId on a repeatable job matters. It prevents the scheduler from stacking a new copy if one is still running, and it lets every instance of the app register the same schedule without creating duplicates. Prefer UTC or an explicit time zone for the pattern: a digest that goes out at 07:00 UTC is not the same as one at 07:00 local, and the difference is a support ticket every daylight-saving change.

Delayed jobs are stored in a sorted set keyed by their run-at time, so a delayed job does not occupy a worker. It sits in Redis until it is due, which makes delays of hours or days cheap.

Concurrency, rate limiting and backpressure

A worker’s concurrency is how many jobs it processes at once. Raising it increases throughput until the worker runs out of CPU, database connections or memory, at which point it makes things worse.

const worker = new Worker("sync", handler, {
  connection,
  concurrency: 10,
  limiter: { max: 50, duration: 1_000 },
});

The limiter caps how many jobs the worker starts per time window. That is the first line of defence against overwhelming a downstream API. If a provider allows 50 requests per second, a worker with concurrency 200 will get you rate-limited; a limiter at 50 per second will not.

Backpressure is what stops the queue itself from growing without bound. If producers add jobs faster than workers can drain them, the queue becomes an ever-growing backlog and its latency becomes hours. Watch queue depth, pause producers above a threshold, and scale workers automatically. A queue that only ever grows is an outage nobody has noticed yet.

Progress and events

Long jobs should report progress so a UI can show a bar and an operator can tell the difference between slow and stuck.

const worker = new Worker("imports", async (job) => {
  const rows = await loadRows(job.data.fileId);

  for (let i = 0; i < rows.length; i += 500) {
    await insertChunk(rows.slice(i, i + 500));
    await job.updateProgress(Math.round((i / rows.length) * 100));
  }

  return { rows: rows.length };
}, { connection });

Events are how the rest of the system observes the queue. A worker emits completed, failed, progress and stalled for the jobs it runs. QueueEvents listens to the same stream from outside the worker, which is how an API process can react to a job finishing without being the one that ran it.

worker.on("progress", (job, progress) => {
  console.log(`job ${job.id} at ${progress}%`);
});

worker.on("completed", (job) => {
  console.log(`job ${job.id} finished`);
});

Graceful shutdown and stalled jobs

A worker killed mid-job leaves that job in an ambiguous state. BullMQ handles this with a lock: while a job is active, the worker renews a lock in Redis. If the worker dies and the lock expires, the job is marked stalled, moved back to waiting and picked up again. This is one of the reasons delivery is at-least-once.

Because a stall leads to a re-run, handle SIGTERM deliberately so in-flight jobs finish instead of being interrupted.

process.on("SIGTERM", async () => {
  await worker.close();   // stop accepting, wait for in-flight jobs
  await connection.quit();
  process.exit(0);
});

Give the deploy a grace period long enough for the slowest job, and cap job timeouts so no single job can outlast it. A handler that can run for an hour should checkpoint its progress so a re-run resumes rather than restarts.

Redis Streams directly

BullMQ is a layer on top of Redis primitives, and sometimes you want the primitives. Streams are the right tool when several consumers must each see every message, when you need to replay history, or when you want acknowledgement without a job framework.

XADD jobs:emails '*' type welcome userId 42

XREADGROUP GROUP workers alice COUNT 10 BLOCK 5000 STREAMS jobs:emails '>'

XACK jobs:emails workers 1760000000000-0

The consumer group tracks which messages have been delivered and which have been acknowledged. A message that is delivered but never acknowledged stays in the group’s pending list, so a crashed consumer does not lose it. XAUTOCLAIM reassigns pending messages from a dead consumer to a live one.

Use streams directly when the data is an event log and the consumers are independent readers. Use BullMQ when the data is a unit of work with a retry policy, a priority and a result. Reimplementing retries, delays and a failed set on top of streams is exactly the work BullMQ already did.

Idempotency: delivery is at-least-once

The most important fact about queues is that delivery is at-least-once, not exactly once. A worker can crash after performing the work but before acknowledging it, and the job will run again. A stall leads to a re-run by design. Exactly-once delivery across a network is effectively impossible, so queues choose at-least-once and push the responsibility to you.

Your handler must be idempotent: running it twice must produce the same end state as running it once. A dedupe key derived from the work, written atomically before the side effect, is the usual technique.

export async function handleCharge(job) {
  const key = `charged:${job.data.orderId}`;
  const first = await connection.set(key, "1", "NX", "EX", 86_400);

  if (first === null) return { skipped: true, reason: "already_processed" };

  await stripe.charges.create(
    { amount: job.data.amount, source: job.data.token },
    { idempotencyKey: job.data.orderId },
  );
}

Notice that the key comes from the order id, not the job id. That makes the handler safe even if the producer enqueues the same logical work twice. Combine the dedupe key with the provider’s own idempotency key, because that key only protects the API call, not the surrounding logic.

Monitoring the queue

A queue is invisible unless you make it visible. Four signals cover most of what you need.

• Queue depth — how many jobs are waiting. Rising depth means workers cannot keep up.
• Oldest waiting job — depth says how many, age says how bad. Ten thousand jobs that clear in a second are fine; ten that have waited an hour are not.
• Failure rate — failures per minute, split by job name. A spike after a deploy points at the change.
• Job duration — a histogram per job name. A rising p95 means a dependency is slowing down.

BullMQ exposes these counts directly, and a QueueEvents listener or an exporter can forward them to your metrics system.

const counts = await queue.getJobCounts(
  "wait", "active", "completed", "failed", "delayed",
);

console.log(counts);

For a visual view, Bull Board or the BullMQ dashboard mounts a small web UI over the same keys. Add a correlation id to every payload and include it in logs so a job can be traced from the request that created it through every retry.

Keeping payloads small and versioned

A queue is a persistent interface between two deploys. A producer running version 1 of the code can write a job that a worker running version 2 must read. This is the same compatibility problem as an API, and it is easy to ignore until a deploy breaks a backlog.

Two habits keep payloads compatible. First, add fields rather than renaming or removing them, and give new fields a sensible default in the handler. A worker that tolerates a missing locale can process jobs enqueued before the field existed. Second, put a version in the payload when the shape can change meaningfully, and branch on it in the handler.

await queue.add("import", { version: 2, importId, mapping });

Keep payloads small for a second reason: Redis holds every waiting job in memory. A payload that embeds a whole database row multiplies across thousands of jobs and makes the queue expensive. Reference data by id and let the worker fetch it. The one exception is a value that must be frozen at enqueue time, such as the recipient of an email or the price quoted to a customer, which belongs in the payload precisely because it should not change.

Composing work with flows

Some jobs are really several jobs. A report might fetch data, render a PDF and email it, and you may want each step retried independently. BullMQ flows express this as a tree of parent and child jobs.

import { FlowProducer } from "bullmq";

const flow = new FlowProducer({ connection });

await flow.add({
  name: "report",
  queueName: "reports",
  data: { reportId },
  children: [
    { name: "fetch", queueName: "reports", data: { reportId } },
    { name: "render", queueName: "reports", data: { reportId } },
  ],
});

A child job runs first, and the parent becomes runnable only when all of its children complete. That gives you fan-out and fan-in without coordinating state in your own database. If a child fails, the parent waits or fails according to the flow options, so the retry policy stays attached to the step that actually broke. Keep flows shallow and explicit; a deep tree of interdependent jobs is harder to reason about than a small pipeline with clear stages.

Choosing Redis queues vs Kafka and RabbitMQ

Redis is the right default when the unit of work is a job: a named task with a payload, a retry policy and an outcome. It is fast, familiar and already in most stacks.

Kafka is the right choice when the data is a log. If several independent consumers must each read every event, if you need to replay history from an offset, or if throughput is measured in millions of messages per second, a partitioned log fits better than a job queue.

RabbitMQ is the right choice when routing is the hard part. Exchanges and binding keys let one message fan out to many queues by pattern, with per-message acknowledgements, priorities and dead-letter exchanges as first-class features. It is heavier to operate than Redis and rewards teams that need that flexibility.

The honest guidance is to start with what you run. A Redis queue is far better than no queue because you were waiting to evaluate brokers. Move when a specific limitation — durability, replay or routing — actually bites.

Best practices

• Keep request handlers to a write and an enqueue, and return 202 Accepted.
• Share one Redis connection per process, with maxRetriesPerRequest: null.
• Derive jobId from the work so duplicate enqueues collapse into one job.
• Make every handler idempotent, because delivery is at-least-once.
• Use exponential backoff with jitter, and set an attempt limit.
• Keep failures by setting removeOnFail: false, and bound completed jobs with removeOnComplete.
• Separate queues by workload so slow jobs cannot starve urgent ones.
• Bound concurrency and add a limiter for any rate-limited dependency.
• Report progress from long jobs so slow and stuck look different.
• Shut down workers on SIGTERM and give deploys a matching grace period.
• Track depth, oldest-job age, failure rate and duration on a dashboard.

Common mistakes

• Assuming a job runs exactly once, then charging a customer twice.
• Using a random job id and stacking duplicate repeatable schedules.
• Retrying a poison message forever and starving the queue.
• Running heavy work in the API process and calling it a queue.
• Setting concurrency so high that the database hits its connection limit.
• Returning a huge object as a job result and bloating Redis.
• Forgetting maxRetriesPerRequest: null and losing jobs on a reconnect.
• Leaving the failed set off every dashboard until the first incident.
• Killing workers with SIGKILL and forcing every in-flight job to stall.
• Putting a whole database row in the payload instead of an id.

Where to go next

The Redis guide covers the store underneath the queue: lists, streams, TTLs, persistence and the single-threaded command model that makes atomic claims possible. For the broader discipline of moving work out of the request path, read Batch Processing. When routing and per-message acknowledgements become the hard part, continue to RabbitMQ, and when you need a replayable log rather than a job queue, the Kafka guide is the next stop.