Node.js Streams

Why streams matter

Loading a large file into memory works until the file is larger than the memory you have. Streams solve this by processing data in chunks: you read a piece, handle it, and move on, so memory stays roughly constant no matter how large the input is.

Streams are everywhere in Node. HTTP request and response bodies, file reads and writes, compression, encryption and many parsers are all streams. Understanding them is what lets you build servers and tools that handle unbounded data without falling over.

The four types of streams

Every stream is one of four kinds:

• Readable — a source you read from. Files, HTTP request bodies, sockets and process.stdin.
• Writable — a destination you write to. Files, HTTP responses, sockets and process.stdout.
• Duplex — both readable and writable, such as a TCP socket.
• Transform — a duplex stream that modifies data as it passes, such as zlib.createGzip() or a CSV parser.

Readable and writable streams connect into pipelines, and transforms sit in the middle.

Reading and writing

The simplest way to consume a readable stream is async iteration.

// read.js
import { createReadStream } from "node:fs";

const stream = createReadStream("access.log", { encoding: "utf8" });

for await (const chunk of stream) {
  process.stdout.write(chunk);
}

For writable streams, call write() and signal the end with end().

// write.js
import { createWriteStream } from "node:fs";

const out = createWriteStream("out.txt");

out.write("first line\n");
out.write("second line\n");
out.end();

The stream buffers writes internally and flushes them efficiently, so you do not have to manage chunks yourself.

Piping and pipeline

A pipeline connects a readable, zero or more transforms, and a writable.

// compress.js
import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";

await pipeline(
  createReadStream("access.log"),
  createGzip(),
  createWriteStream("access.log.gz"),
);

Always prefer pipeline over manual .pipe() chaining. pipeline forwards errors from every stage, destroys the streams on failure and returns a promise, which makes error handling straightforward. Manual .pipe() does not propagate errors, so a failure upstream can leave the pipeline hanging and the destination open.

Backpressure

Streams have internal buffers. When you write faster than the destination can consume, the buffer fills and write() returns false.

// backpressure.js
function writeAll(stream, chunks) {
  return new Promise((resolve, reject) => {
    let i = 0;
    const next = () => {
      while (i < chunks.length) {
        const ok = stream.write(chunks[i++]);
        if (!ok) {
          stream.once("drain", next);
          return;
        }
      }
      stream.end(resolve);
    };
    stream.on("error", reject);
    next();
  });
}

Waiting for drain before writing more is how a producer respects a slow consumer. pipeline and pipe handle this automatically, which is another reason to use them.

Transform streams

A transform stream applies a function to each chunk. You can build your own.

// upper.js
import { Transform } from "node:stream";

const upper = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  },
});

process.stdin.pipe(upper).pipe(process.stdout);

Node ships useful transforms in node:zlib (gzip, deflate), node:crypto (ciphers and hashes) and many libraries, such as CSV and JSON parsers.

Object mode

By default streams carry bytes. Object mode lets them carry JavaScript objects instead, which is ideal for structured pipelines.

// object-mode.js
import { Readable, Transform } from "node:stream";

Readable.from([{ id: 1 }, { id: 2 }])
  .pipe(
    new Transform({
      objectMode: true,
      transform(record, _encoding, callback) {
        callback(null, { ...record, seen: true });
      },
    }),
  )
  .on("data", console.log);

In object mode, highWaterMark counts objects rather than bytes. It is how database row streams, log processors and ETL pipelines are built.

Common patterns

• Copy a file: pipeline(createReadStream(src), createWriteStream(dest)).
• Compress or encrypt: insert createGzip() or a cipher transform.
• Stream an HTTP response: pipe a file or a query result straight to res.
• Parse line-delimited data: use a transform that splits on newlines.
• Upload progress: count bytes in a transform as they pass.

Best practices

• Prefer pipeline over .pipe() for error handling and cleanup.
• Stream large files and responses instead of buffering them.
• Respect backpressure; never ignore write() returning false.
• Tune highWaterMark only after measuring.
• Use object mode for structured data.
• Handle error on every stream you create.
• Destroy streams on failure so file descriptors are released.

Common mistakes

• Using readFile for large or unbounded data.
• Chaining .pipe() and losing errors.
• Ignoring backpressure and buffering unbounded data.
• Forgetting that a transform callback must be called exactly once.
• Mixing encodings and producing corrupted output.
• Leaving streams open after an error.

Where to go next

Streams are how Node handles data at scale. Put them to work with the File System guide and understand the scheduling underneath in the Event Loop guide. Then rewrite one readFile call as a stream and watch memory flatten.