CI/CD Pipelines

What CI and CD actually mean

Continuous integration is the practice of merging every developer’s work into a shared branch frequently and verifying it automatically. Each merge triggers a build, a lint pass and the test suite on a machine that nobody develops on. The point is to find integration problems in minutes, while the change is small, instead of in a painful merge at the end of a long branch.

Continuous delivery extends that idea: the main branch is always in a releasable state, and the pipeline can deploy it to production at any moment. The artifact is built and tested on every change; a human decides when to release. Continuous deployment removes that human decision, sending every change that passes the gates straight to production.

The difference between delivery and deployment is the approval gate, and it is the only difference. Everything before it — the build, the tests, the artifact — is identical.

Teams often claim to “do CI/CD” while doing neither. Continuous integration requires that the build runs on a shared, authoritative machine, not just on a laptop before a commit. Continuous delivery requires that the artifact produced by the pipeline is the one that ships, not something rebuilt by hand at release time. If either is missing, the loop is open and the benefits leak out.

The anatomy of a pipeline

Every pipeline, from a ten-line GitHub Actions file to a thousand-job enterprise setup, is built from the same vocabulary.

• Trigger — the event that starts a run: a push, a pull request, a tag, a schedule or a manual dispatch.
• Workflow — the file that declares triggers and jobs. In GitHub Actions it lives in .github/workflows/.
• Job — a unit of work that runs on one runner. Jobs run in parallel unless you declare dependencies with needs.
• Step — a single command or reusable action inside a job. Steps run in order and the job stops at the first failure.
• Runner — the machine that executes a job. Hosted runners are disposable; self-hosted runners are machines you manage.
• Artifact — a file produced by a job and stored for later jobs or for humans: a binary, a bundle, a test report.
• Cache — a directory restored between runs to avoid repeating expensive work such as installing dependencies.
• Environment — a named target such as staging or production with its own secrets, protection rules and URL.
• Secret — an encrypted value injected into a job at runtime and masked in logs.
• Status check — the pass or fail result reported back to the commit, which branch protection can require.

Once these words are clear, any CI system becomes readable. Jenkins, GitLab CI, CircleCI and GitHub Actions all use the same concepts with different spellings.

A GitHub Actions workflow, line by line

The smallest useful workflow checks out the code, installs dependencies and runs the tests.

name: ci

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm

      - run: npm ci
      - run: npm test

on declares the triggers. A push to main and any pull request both start a run. jobs.test runs on a fresh ubuntu-latest runner. uses runs a reusable action; run executes a shell command. npm ci installs exactly what the lockfile pins, which is what makes the run reproducible. cache: npm tells the setup action to cache the npm directory, keyed by the lockfile.

Add concurrency to cancel superseded runs, which saves both time and money on active branches:

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

When a developer pushes three commits in a row, only the newest run survives. The older ones are cancelled, and the pull request shows a single current status instead of a queue of stale ones.

Build once, deploy the same artifact

The most valuable habit in a pipeline is building exactly once. The pipeline compiles, tests and packages the application one time, and every environment from staging to production runs that same output.

The alternative — rebuilding for each environment — seems harmless and is not. Different builds can resolve different dependency versions, pick up different base image digests, or embed a different NODE_ENV. Staging then tests a binary that production will never run, and “it passed staging” proves nothing.

Two rules make build-once practical. First, tag the artifact by commit, never by a mutable name like latest:

docker build -t ghcr.io/me/app:$GIT_SHA .
docker push ghcr.io/me/app:$GIT_SHA

Second, inject configuration at runtime. The image should not know whether it is running in staging or production. Database URLs, feature flags and API keys arrive as environment variables when the container starts. The build is identical everywhere; only the environment differs.

To promote, reference the immutable digest rather than the tag, so even a retagged image cannot drift:

kubectl set image deploy/app \
  app=ghcr.io/me/app@sha256:...

This is the foundation that makes rollbacks trivial. The previous release is simply the previous digest, still sitting in the registry, ready to deploy in seconds.

Tests, lint and typecheck as gates

A gate is a check that must pass before the pipeline continues. The essential gates are fast and cheap: linting, type checking and the unit test suite. They run on every pull request, and they stop the artifact from being built when they fail.

Order them from fastest to slowest. Lint and typecheck usually finish in seconds and catch a large class of mistakes. Unit tests are next. Slower integration and end-to-end tests can run in parallel or only on the main branch.

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npm test

Two jobs run in parallel, each on its own runner. Lint fails in twenty seconds without waiting for the test suite. The trade-off is that both jobs install dependencies; a shared cache keeps that cheap.

Gates only matter if they are required. On the forge, mark the checks as required in branch protection so a pull request cannot merge while any of them is red.

Matrices and parallelism

A matrix runs one job definition across several combinations of inputs. It is how you test multiple Node versions and operating systems without duplicating YAML.

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        node: [20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: npm
      - run: npm ci
      - run: npm test

This expands to four jobs. fail-fast: false lets the others finish even if one fails, which is useful when you want to know whether the failure is version-specific.

Parallelism is not free. Each job provisions a runner and installs dependencies, so a wide matrix costs more time and money than a narrow one. Test the versions you actually support, not every version that exists.

For large suites, shard the tests themselves: pass a shard index and total to the test runner so each of four jobs runs a quarter of the tests. The suite finishes in roughly a quarter of the time.

Secrets and environment variables

Secrets are encrypted values the pipeline injects at runtime. They belong in the platform’s secret store, never in the repository, and never in the image.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          API_KEY: ${{ secrets.API_KEY }}

The platform masks known secret values in logs, but masking is best-effort. It cannot catch a secret that has been transformed, base64-encoded or printed by a tool that reformats it. The safe rule is simple: never print a secret. Do not run env in a debug step, do not echo a token, and do not pass a secret to a command that logs its arguments.

Scope secrets to the environment that needs them. A staging deploy has no reason to hold the production database password. On GitHub, environment secrets are only available to jobs that declare that environment, and can require approval first.

For cloud providers, prefer OIDC over long-lived keys. The pipeline exchanges a short-lived identity token for temporary credentials, so there is no static secret to leak or rotate.

permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/deploy
      aws-region: eu-west-1

The token is valid for minutes and scoped to one role. If it leaks, the blast radius is a single run.

One warning deserves emphasis: pull requests from forks run untrusted code. Never expose secrets to that code. Use pull_request rather than pull_request_target for anything that checks out and builds contributor code, and require approval before running workflows from first-time contributors.

Environments, approvals and protected branches

An environment is a named deployment target with its own secrets, protection rules and URL. It is the mechanism that keeps production distinct from everything else.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://app.example.com
    steps:
      - run: ./deploy.sh

Environments can require a manual approval before a job runs, restrict which branches may deploy to them, and limit how long a deployment waits before timing out. This is where continuous delivery’s “human button” lives: the pipeline runs automatically up to the gate, and a reviewer releases it.

Protected branches complement environments. On the main branch, require a pull request, require passing status checks, require review, and forbid force pushes. Together they mean that the only way into production is through the pipeline, which is the whole point.

Deployment strategies and rollbacks

How a new version replaces an old one is a design decision with real consequences for users.

Rolling deployment replaces instances a few at a time. Capacity dips slightly during the swap, but no extra infrastructure is needed. It is the default on most platforms and the right starting point.

Blue-green deployment runs two complete environments. Traffic points at blue while green receives the new version; once green is healthy, a single routing change flips all traffic. Rollback is flipping back, which is nearly instant. The cost is running two environments at once.

Canary deployment sends a small slice of traffic — one percent, then five, then fifty — to the new version and watches error rates and latency before proceeding. It catches problems that a health check cannot, at the cost of more complex routing and monitoring.

Whatever the strategy, the deployment must be reversible. Because the previous artifact is immutable and still in the registry, a rollback is a redeploy of the previous digest, not a rebuild:

kubectl rollout undo deploy/app

Two details make rollbacks safe. Database migrations should be backward compatible for at least one release, so the old code can still run against the new schema. And feature flags let you disable new behaviour without deploying anything at all, which is the fastest rollback there is.

Artifact registries and container images

An artifact registry stores the build output: container images, npm packages, binaries, tarballs. It is the handoff point between building and deploying, and it should be immutable.

Container images are the most common artifact because they carry their runtime. A registry such as GitHub Container Registry, Amazon ECR or Docker Hub stores layers, and the pipeline pushes to it once per build. Deploys then pull the exact digest.

- uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

The cache-from and cache-to options persist Docker layer caching across runs, so an unchanged dependency layer is not rebuilt from scratch every time.

Keep a retention policy. Registries accumulate gigabytes quickly, and deleting old images is part of keeping the pipeline fast. Keep at least the last few releases so rollback stays possible.

Status checks and branch protection

A status check is the pipeline’s verdict on a commit. The forge attaches it to the pull request, and branch protection decides whether it is advisory or mandatory.

Configure the main branch to require:

• A pull request before merging, with at least one approval.
• Passing status checks, including lint, typecheck and tests.
• A branch up to date with main before merging, so the checks ran against the code that will actually land.
• No force pushes and no deletions.

The result is that the main branch is always green. Every commit on it has passed the same gates, and every deploy from it is a known-good artifact.

Keep the required list short. Requiring a slow end-to-end suite on every pull request makes developers wait; requiring it on the merge queue or main branch gives the same safety without the friction.

Caching and incremental builds

Caching is the difference between a two-minute pipeline and a twelve-minute one. The principle is to never repeat work whose inputs have not changed.

The most valuable cache is dependencies. Key it on the lockfile hash so it invalidates exactly when dependencies change:

- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: npm-

restore-keys allows a partial match: even when the exact key misses, the most recent cache is restored and updated incrementally, which is much faster than a cold install.

Build tools add their own caches. TypeScript’s incremental build, Vite’s dependency pre-bundling, and Docker’s layer cache all benefit from being persisted between runs. For Docker, use the registry or the CI provider’s cache backend rather than the local daemon, which is discarded with the runner.

A cache is an optimisation, never a source of truth. If a cache is corrupt or stale, the run must still be correct. Build tools that trust a cache blindly can produce wrong output, so key caches conservatively and treat a cache miss as normal.

Pipelines for monorepos

A monorepo holds many packages in one repository. A naive pipeline rebuilds and retests all of them on every change, which gets slower as the repository grows.

The fix is change detection. Path filters and dependency graphs tell the pipeline which packages are affected by a diff, and only those are built.

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm turbo run test --filter='...[origin/main]'

The filter selects packages changed since main plus everything that depends on them. A change to a shared utility triggers its dependents; a change to a leaf package triggers only itself.

Add a remote cache so compiled output and test results are shared across machines and runs. If a package’s inputs are unchanged, its build is restored rather than recomputed. In a monorepo, this is often a larger win than dependency caching.

Keep the pipeline honest: a change to the root lockfile or shared config should still run the full suite. Selective testing is an optimisation that must never let a broken package slip through.

Pipeline security

The pipeline holds credentials to production, which makes it a high-value target. Treat workflow files as security-sensitive code and review changes to them carefully.

Pin actions to a version. A tag like @v4 is a moving pointer; a full commit SHA is immutable. Pinning to a SHA is the strictest option and prevents a compromised tag from running malicious code in your pipeline.

Grant the least privilege. Set permissions explicitly on each workflow and job, defaulting to read-only, and add write scopes only where required.

permissions:
  contents: read

The default GITHUB_TOKEN is often broader than any job needs. Narrowing it means a compromised step cannot push commits or publish packages.

Prefer OIDC over static keys, as described above. Short-lived credentials scoped to one role remove the long-lived secret that attackers most want.

Do not run untrusted code with secrets. Pull requests from forks are the classic vector. Use pull_request for build and test, require approval for first-time contributors, and never check out and execute fork code in a workflow that has access to secrets.

Review third-party actions. An action is a dependency that runs with your credentials. Prefer actions from the forge itself or well-known publishers, and audit the rest.

Best practices

• Build the artifact exactly once and promote it through every environment.
• Tag artifacts by commit SHA or digest, never only by latest.
• Inject configuration at runtime; keep the build identical everywhere.
• Run lint and typecheck before the slower test suite so failures are fast.
• Make the important checks required in branch protection.
• Cache dependencies keyed by the lockfile hash, with restore keys.
• Scope secrets to environments and prefer short-lived OIDC credentials.
• Never print secrets, and never expose them to untrusted pull requests.
• Pin third-party actions and set least-privilege token permissions.
• Use concurrency to cancel superseded runs.
• Keep the pipeline fast; a slow one gets bypassed.
• Make rollbacks a redeploy of the previous digest.

Common mistakes

• Rebuilding the image differently for each environment and calling it promotion.
• Baking environment configuration into the image at build time.
• Using latest as the only tag and losing traceability.
• Echoing secrets in a debug step and assuming masking catches everything.
• Exposing secrets to workflows triggered by fork pull requests.
• Referencing third-party actions by a moving branch.
• Leaving the default token permissions wide open.
• Marking every optional check as required and making merges crawl.
• Letting flaky tests stay red until nobody reads the results.
• Caching aggressively without a key that invalidates on real input changes.
• Rebuilding everything in a monorepo on every change.
• Having no rollback path other than reverting a commit and waiting.

Where to go next

A pipeline ends by deploying an artifact, so the natural next step is Cloud Platforms, where the release, health checks and rollbacks actually happen. The artifact is usually a container, and the Docker guide covers the images and multi-stage builds the pipeline produces. Most runners are Linux machines, so the Linux guide explains the shell and tools your steps depend on, and Node.js grounds the runtime and toolchain being built.