Batch Processing

What is batch processing?

Batch processing is the practice of taking work that is too slow, too fragile or too bursty to do while a user waits, and handing it to a separate process that runs it on its own schedule. The user’s request does the minimum it must — validate input, write a row, enqueue a job — and returns. Everything else happens out of band.

The name comes from mainframe-era jobs that processed a stack of records in one run, and the idea has not changed. Instead of a tape, you have a queue. Instead of a nightly window, you have workers pulling continuously. What is constant is the separation: the thing that accepts work and the thing that performs it are different, and a buffer sits between them.

This separation is the whole point. It lets the request path stay fast and predictable while the slow path takes as long as it needs, retries when it fails, and scales on a different curve.

Why heavy work does not belong in a request

A synchronous HTTP request is a bad place for slow work, for three reasons that compound.

First, timeouts. Proxies, load balancers and clients all impose deadlines. A request that renders a 200-page PDF, calls a flaky third-party API and sends an email can easily exceed them. When it does, the client sees an error even though the work may have half-completed on the server.

Second, a blocked event loop. Node runs JavaScript on a single thread. A synchronous CPU-heavy task — image processing, a large JSON parse, a cryptographic loop — stops every other request from being served while it runs. Even asynchronous work holds resources: an open database connection, a file handle, memory for the response.

Third, retries are impossible. If the mail provider returns a 503, an inline handler has no good options. It can fail the whole request and make the user retry, or swallow the error and lose the email. A queue gives the same failure a third answer: try again later, automatically, without the user knowing.

app.post("/reports", async (req, res) => {
  const report = await db.report.create({ data: { userId: req.user.id } });

  await reportsQueue.add("generate", { reportId: report.id });

  res.status(202).json({ id: report.id, status: "queued" });
});

The 202 Accepted status is the honest answer here: the server has accepted the request but has not finished the work. It is the shape of every well-behaved batch endpoint.

The anatomy of a job

A job is a small, self-describing record. Most queues store something like this:

{
  "id": "welcome:user_42",
  "name": "welcome",
  "data": { "userId": "user_42", "to": "[email protected]", "template": "welcome" },
  "status": "queued",
  "attemptsMade": 0,
  "maxAttempts": 5,
  "runAt": 1760000000000,
  "createdAt": 1759999100000
}

Every field earns its place. The id makes the job addressable and, when it is derived from the work rather than random, gives you deduplication for free. The name routes the job to a handler. The payload carries everything the worker needs — and nothing more, because a payload that points at a database row is smaller and fresher than one that copies it. The status tracks the job through its lifecycle. attemptsMade and maxAttempts drive retries. runAt schedules delayed and recurring work.

The payload should be a snapshot of intent, not a live object. If a user updates their email between enqueue and execution, the job should still send to the address it was created with. Storing the whole user record, though, is a mistake: it bloats the queue and goes stale. Store ids and the few values that define the work.

Job payloads and versioning

A queue is a persistent interface between two deploys. A producer running version 1 of the code can write a job that a worker running version 2 must read. This is the same compatibility problem as an API, and it is easy to ignore until a deploy breaks a backlog.

Two habits keep payloads compatible. First, add fields rather than renaming or removing them, and give new fields a sensible default in the handler. A worker that tolerates a missing locale can process jobs enqueued before the field existed. Second, put a version in the payload when the shape can change meaningfully:

await queue.add("import", { version: 2, importId, mapping });

The handler then branches on version and knows exactly how to interpret the rest. This costs almost nothing and turns a class of post-deploy incidents into a lookup table.

Keep payloads small. A queue stores every waiting job, so a payload that embeds a large object multiplies across thousands of rows and slows every scan. Reference the data by id and let the worker fetch it. The one exception is a value that must be frozen at enqueue time — the recipient of an email, the price quoted to a customer — which belongs in the payload precisely because it should not change.

Queues, cron and event-driven triggers

Not every background task needs a queue, and picking the wrong trigger makes a simple problem complicated.

Cron runs a handler on a wall-clock schedule: every night at 02:00, every Monday at 09:00. It is the right tool for periodic reconciliation, report generation and cleanup. Its weakness is that it has no concept of work units. A cron job that takes longer than its interval will overlap with itself, and a missed run is simply missed.

Event-driven triggers react to something that just happened: a webhook arrived, a row was inserted, a file landed in storage. They are immediate and natural, but they offer no buffering, no retry and no backpressure. A webhook handler that does real work is just an inline request in disguise.

Queues sit between the two. A job is a durable, retryable unit of work that can be scheduled for now or for later. Most production systems use all three: cron enqueues a fan-out job, events enqueue jobs in response to user actions, and workers drain the queue. The rule of thumb is that cron and events decide when to work, and the queue decides how to work.

Producers, workers and the queue

Three roles make up the system, and keeping them distinct keeps it maintainable.

The producer is any code that adds a job. It knows the job name and payload shape and nothing else. It should be fast, and it should be safe to call twice — if the producer itself retries after a timeout, you do not want two jobs.

The queue is a durable, ordered store. Redis with BullMQ, Amazon SQS, RabbitMQ and Google Cloud Tasks all play this role. The queue persists jobs across restarts, hands them out atomically, tracks attempts and moves exhausted jobs aside. You can also build one on a database table, which is a reasonable start when volume is low and a liability once it is not.

The worker is a long-running process that pulls jobs and runs handlers. It is separate from the API for good reasons: it can be deployed on CPU-optimised hardware, scaled to the queue depth, and restarted without dropping requests. In BullMQ the separation is explicit:

import { Worker } from "bullmq";
import { connection } from "./queue.js";

const worker = new Worker(
  "emails",
  async (job) => {
    await sendEmail(job.data);
  },
  { connection, concurrency: 10 },
);

One process can host several workers for different queues, and one queue can be served by many worker processes. The queue is the only shared state, which is exactly why it scales horizontally so cleanly.

Choosing a broker

The three brokers you will meet most often sit at different points on a spectrum of features and operational weight.

Redis with BullMQ is the default for Node teams. Redis is likely already in the stack, the client is mature, and BullMQ adds delayed jobs, repeatable jobs, priorities, rate limiting, retries and a UI. The catch is that Redis is primarily an in-memory store, so durability depends on how you configure persistence. A job acknowledged but not yet written to disk can be lost if the instance dies.

Amazon SQS is fully managed and effectively unlimited in throughput. You get durability and availability without running anything, and pay per request. In exchange you give up some ergonomics: delayed delivery is capped, there is no built-in scheduler for cron-style jobs, and the API is lower level than a job framework.

RabbitMQ is the most flexible. Exchanges and routing keys let one message fan out to many consumers, and per-message acknowledgements, priorities and dead-letter exchanges are first-class. It is heavier to operate than Redis and has a steeper learning curve, but it fits complex routing.

The honest guidance is to start with what you already run. A queue on Redis is far better than no queue because you were waiting to evaluate brokers. Move when a specific limitation — durability, throughput or routing — actually bites.

Idempotency and at-least-once delivery

The single most important fact about job queues is that delivery is at-least-once, not exactly once. A worker can crash after performing the work but before acknowledging it, and the queue will redeliver the job. A network blip can make an acknowledgement disappear. The work runs again.

This is not a bug to work around; it is the contract. Exactly-once delivery across a network is effectively impossible, so queues choose at-least-once and push the responsibility to you. Your handler must be idempotent: running it twice must produce the same end state as running it once.

There are three practical techniques.

Natural idempotency. Some operations are already safe to repeat. Setting a user’s status to active twice is the same as once. Deleting a row by id the second time is a no-op. Prefer these when you can.

A dedupe key. Write a marker keyed by the work before performing it, and skip if the marker already exists. A unique constraint or a Redis SET NX makes the check atomic, so two concurrent workers cannot both win.

export async function handleCharge(job) {
  const key = `charged:${job.data.orderId}`;
  const inserted = await connection.set(key, "1", "NX", "EX", 86_400);
  if (inserted === null) return { skipped: true };

  await stripe.charges.create(
    { amount: job.data.amount, source: job.data.token },
    { idempotencyKey: job.data.orderId },
  );
}

Provider idempotency keys. Payment gateways and many other APIs accept an idempotency key. Pass the job’s stable id and the provider will return the original result instead of charging twice. Always combine this with your own dedupe, because the key only protects the call, not the surrounding logic.

Notice that the dedupe key is derived from the work — the order id — rather than the job id. That is deliberate: it makes the handler safe even if the producer enqueues the same logical work twice.

Retries with exponential backoff and jitter

Transient failures are normal. A database fails over, an API rate-limits you, a container is rescheduled. Retrying is the correct response, but retrying immediately is not.

Exponential backoff increases the delay with each attempt: roughly 2s, 4s, 8s, 16s, 32s. This gives a struggling dependency time to recover instead of being hammered. Jitter adds a random amount to each delay so that many jobs that failed together do not retry together, which would recreate the spike that caused the failure.

Most queues support this declaratively. In BullMQ it is a property of the job:

await queue.add(
  "sync",
  { accountId },
  {
    attempts: 5,
    backoff: { type: "exponential", delay: 2_000 },
  },
);

That produces delays of about 2s, 4s, 8s, 16s and 32s, with the queue’s own jitter applied. When you implement backoff by hand, add the jitter yourself and cap the delay so a job does not sleep for a day:

function nextDelay(attempt: number, base = 1_000, cap = 60_000) {
  const exponential = Math.min(cap, base * 2 ** attempt);
  const jitter = Math.random() * exponential * 0.5;
  return Math.round(exponential + jitter);
}

Set an attempt limit, and decide deliberately what happens when it is reached. Some jobs should retry forever at a low frequency — a reconciliation task, for instance — but most should stop and ask for help.

Dead-letter queues and poison messages

A poison message is a job that fails every time it runs: malformed data, a bug in the handler, a missing referenced row. Because it always fails, it consumes a worker slot on every attempt and can starve healthy work. Retrying it forever is worse than not retrying at all.

The answer is a dead-letter queue (DLQ). After the attempt limit is reached, the queue moves the job — payload, attempts and last error — to a separate holding area. Workers never touch it, so it cannot starve anything, and an operator can inspect it, fix the cause and replay it.

const worker = new Worker("emails", handler, { connection });

worker.on("failed", async (job, err) => {
  if (job && job.attemptsMade >= (job.opts.attempts ?? 1)) {
    await deadLetter.add("failed-email", {
      payload: job.data,
      error: err.message,
      failedAt: new Date().toISOString(),
    });
    await alerting.notify(`job ${job.id} exhausted retries`);
  }
});

Treat the DLQ as an operational surface, not a graveyard. Alert when its depth grows, give it a dashboard, and build a replay path. A DLQ nobody reads is where bugs go to hide.

Concurrency, rate limiting and backpressure

A worker’s concurrency is how many jobs it processes at once. Raising it increases throughput until the worker runs out of CPU, database connections or memory, at which point it makes things worse. The right value is the largest number that keeps every dependency comfortably below its limit.

Concurrency is also the first line of defence against a thundering herd. If a downstream API allows 50 requests per second, a worker with concurrency 200 will overwhelm it. Many queues offer a rate limiter for exactly this:

const worker = new Worker("sync", handler, {
  connection,
  concurrency: 10,
  limiter: { max: 50, duration: 1_000 },
});

Backpressure is what keeps the queue itself from growing without bound. If producers add jobs faster than workers can drain them, the queue becomes an ever-growing backlog and its latency becomes hours. Options include pausing producers when depth crosses a threshold, rejecting low-priority jobs, and scaling workers automatically. A queue that only ever grows is an outage that has not been noticed yet.

Keep queues separate by workload. A slow nightly import and a time-sensitive password reset should not share a line.

Batching database writes

The database is usually the bottleneck in a batch job, and the most common cause is talking to it one row at a time. Each round trip has fixed overhead — network, parsing, planning — that dwarfs the cost of the row itself. A loop of single inserts spends most of its time waiting.

A multi-row insert moves the same data in one statement:

const values = chunk
  .map((_, n) => `($${n * 3 + 1}, $${n * 3 + 2}, $${n * 3 + 3})`)
  .join(",");

await pool.query(
  `INSERT INTO orders (user_id, status, total_cents)
   VALUES ${values}
   ON CONFLICT (external_id) DO UPDATE
     SET status = EXCLUDED.status`,
  chunk.flatMap((r) => [r.userId, r.status, r.totalCents]),
);

The ON CONFLICT ... DO UPDATE clause turns the insert into an upsert, which is what makes a bulk write idempotent. Re-running the job updates existing rows instead of creating duplicates, so a retry is safe.

Two cautions. Keep chunks bounded — a few hundred to a few thousand rows — because a parameterised statement has a limit on parameters and a very large statement holds locks and memory longer. And wrap a chunk in a transaction if the rows must land together, but keep the transaction short so it does not block other writers.

For very large loads, a dedicated bulk path such as Postgres’s COPY is faster still, as covered in the PostgreSQL guide.

Chunking large datasets

A batch job that processes “all users” cannot load them all into memory. The fix is to chunk the work: process a bounded page, commit, then fetch the next. This keeps memory flat and lets the job resume from where it stopped.

Keyset pagination is the robust way to do it. Instead of OFFSET, which gets slower as it grows and can skip or repeat rows when data changes underneath, you remember the last key you saw:

let cursor: string | null = null;

while (true) {
  const batch = await pool.query(
    `SELECT id, email FROM users
     WHERE ($1::text IS NULL OR id > $1)
     ORDER BY id
     LIMIT 1000`,
    [cursor],
  );

  if (batch.rowCount === 0) break;

  await processBatch(batch.rows);
  cursor = batch.rows[batch.rows.length - 1].id;
}

Each chunk is independent, so a crash mid-run only loses the current chunk, and the job can be resumed by passing the last cursor. This pairs naturally with the queue: enqueue one job per chunk so a single failure does not force a restart of the entire run.

Scheduling recurring jobs

Recurring work — daily digests, cleanup, reconciliation — is best expressed as a repeatable job rather than a cron entry that calls an HTTP endpoint. The queue then owns the schedule, the overlap protection and the retry policy.

await reportsQueue.add(
  "daily-digest",
  { region: "eu" },
  {
    repeat: { pattern: "0 7 * * *" },
    jobId: "daily-digest:eu",
    attempts: 3,
  },
);

The stable jobId matters: it prevents the scheduler from stacking a new copy if one is still running, and it lets every instance of the app register the same schedule without creating duplicates. A distributed lock around the actual work is still wise for jobs that must never run twice concurrently.

Prefer UTC for schedules and make the job aware of time zones when it formats output. A digest that goes out at 07:00 UTC is not the same as one at 07:00 local, and the difference is a support ticket every daylight-saving change.

Observability

A background job is invisible unless you make it visible. Four signals cover most of what you need.

• Queue depth — how many jobs are waiting. A rising depth means workers cannot keep up, which is the earliest warning of an outage.
• Job duration — a histogram per job name. A p95 that climbs over time signals a dependency slowing down.
• Failure rate — failures per minute, split by job name. A spike after a deploy points straight at the change.
• Age of the oldest waiting job — depth says how many, age says how bad. Ten thousand jobs that clear in a second are fine; ten that have waited an hour are not.

Add a correlation id to every job payload and include it in logs, so a job can be traced from the request that created it through every retry. Without it, debugging a worker failure means grepping timestamps and guessing.

await queue.add("import", { importId, correlationId: req.id });

Expose the metrics on the same dashboard as your API, and alert on depth and oldest-job age rather than on individual failures, which are expected.

Graceful shutdown

A worker killed mid-job leaves that job in an ambiguous state. The queue will eventually redeliver it, which is correct but wasteful, and a hard kill can interrupt a database transaction at the worst moment.

Handle SIGTERM and close the worker deliberately:

process.on("SIGTERM", async () => {
  await worker.close();
  await connection.quit();
  process.exit(0);
});

worker.close() stops accepting new jobs and waits for in-flight ones to finish. Pair it with a deployment grace period long enough for the slowest job, and cap job timeouts so no single job can outlast it. If a job is genuinely long-running, make it checkpoint its progress so it can resume rather than restart.

The same discipline applies to the connection: close the database pool and the broker client so the process exits cleanly instead of hanging on open sockets.

Best practices

• Keep request handlers to a write and an enqueue; return 202 when work is deferred.
• Make every handler idempotent, because delivery is at-least-once.
• Derive dedupe keys from the work, not from a random job id.
• Use exponential backoff with jitter and cap the delay.
• Set an attempt limit and route exhausted jobs to a dead-letter queue.
• Separate queues by workload so slow jobs cannot starve urgent ones.
• Bound concurrency to what your database and downstream APIs can take.
• Batch database writes with multi-row inserts or upserts, in bounded chunks.
• Page large datasets with keyset pagination, not OFFSET.
• Track queue depth, job duration, failure rate and oldest-job age.
• Shut down workers gracefully and give deploys a matching grace period.

Common mistakes

• Doing slow third-party calls inside the request and calling it fine because it works locally.
• Assuming a job runs exactly once, then charging a customer twice.
• Retrying immediately with no backoff and amplifying the original failure.
• Retrying a poison message forever and starving the queue.
• Running one giant job that processes every row in a single transaction.
• Inserting rows one statement at a time and blaming the database.
• Setting concurrency so high that the database hits its connection limit.
• Scheduling recurring jobs with a random id and stacking duplicates.
• Never looking at the dead-letter queue.
• Killing workers with SIGKILL and losing in-flight work.
• Leaving the queue depth off the dashboard until the first incident.

Where to go next

A queue is only as good as the store behind it, so the Redis guide is the natural next read for the broker most Node teams use. The Caching guide shows how to avoid doing the work at all, and Connection Pooling explains how to keep a fleet of workers from exhausting your database. When the job is a bulk write, the PostgreSQL guide covers COPY, upserts and the transaction shapes that make it fast.