When CI takes more than 20 minutes, behaviour changes. People stop pushing small commits, they batch unrelated changes to "save a run", they context-switch while waiting, and flaky failures cost half an hour each. Our main pipeline had crept up to 34 minutes. Over six weeks we brought it down to 14, and the number of pull requests per engineer per week went up by about a fifth.

None of the changes were exotic. This guide is the order in which we did them, because order matters: measuring first stopped us from optimising things that were not on the critical path.

1. Measure the critical path

Total CPU minutes and wall-clock time are different problems. Wall-clock time is set by the longest chain of dependent jobs. Make that chain visible:

checkout+deps (3m) -> build (6m) -> unit tests (11m) -> integration tests (9m) -> image (4m)
                                   lint (2m)           e2e (14m, waits for image)

Export job timings from your CI provider's API into a spreadsheet or dashboard for the last 100 runs and look at medians and p90 per job. In our case, unit tests and the end-to-end suite were the critical path, and the e2e suite waited on an image build it did not really need until its last step.

2. Remove waiting: restructure dependencies

Jobs often depend on each other out of habit. We changed three things:

  • Lint and type checks run in parallel with build, not after it.
  • Integration tests start from the build artifact, not after unit tests pass. Failing fast matters less than total time when both usually pass.
  • The e2e suite builds what it needs from source in its own job, instead of waiting for the production image.

That alone saved 9 minutes of wall-clock time without making anything faster.

3. Cache the expensive, stable things

Caching works when the cache key changes rarely and the cached work is expensive:

Cache Key Saved
Package manager download cache Lockfile hash 2-3 min
Build tool cache (Gradle, Go build, Turborepo) Lockfile + tool version, with restore keys 3-5 min
Docker layers Per image and branch, mode=max for multi-stage 4-6 min
Test databases Migration files hash, as a prebuilt image 1-2 min

Two rules: always key on the lockfile, never only on the branch, and check cache hit rates. A cache with a 20 percent hit rate costs upload time and gives little back.

4. Split tests by timing, not by file count

Running tests in parallel shards is the biggest single win for most suites, but naive splitting by file count gives uneven shards: one shard gets the slow integration files and becomes the new bottleneck. Split by historical timing instead. Most test runners can emit JUnit XML with durations; store it as an artifact on the main branch and use it to balance shards.

strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4, 5, 6]
steps:
  - run: |
      ./ci/split-tests --timings timings/junit.xml --shard ${{ matrix.shard }} --total 6 > files.txt
      ./ci/run-tests $(cat files.txt) --junit-out junit-${{ matrix.shard }}.xml

Six shards took unit tests from 11 minutes to just under 2.5, slightly worse than a perfect 6x because each shard pays setup cost. Past a certain shard count, setup dominates, so measure where your curve flattens.

5. Build and test only what changed

In a monorepo, running everything on every change is the default and the most wasteful option. Tools like Nx, Turborepo, Bazel, Pants or Gradle's build cache know the dependency graph and can compute affected projects:

npx nx affected -t lint test build --base=origin/main --head=HEAD

Two safety rules: always run the full suite on the main branch after merge, and treat changes to shared config (lockfiles, CI files, base images) as "everything affected".

6. Fix flaky tests and remove sleeps

Flaky tests cost far more than their runtime because they cause full reruns. We tracked tests that passed on retry, quarantined the worst offenders into a non-blocking job with an owner and a deadline, and searched the suite for fixed sleeps. Replacing sleep 10 waits with polling for readiness saved two minutes in integration tests alone.

7. Size runners deliberately

Larger runners are not always faster, but CPU-bound steps like compilation and parallel tests scale well. We benchmarked our build on 2, 4, 8 and 16 vCPU runners. Going from 2 to 8 cut build time by 55 percent at roughly the same cost per run, because the job finished faster. Going to 16 helped very little. For IO-heavy jobs, faster disks mattered more than CPUs.

8. Keep it fast

Pipelines get slower one reasonable change at a time. We publish median and p90 pipeline duration for the main branch on the team dashboard, and any pull request that adds more than a minute to the critical path needs a sentence in its description explaining why. Twice a year we repeat the measurement from step 1, because the critical path moves as the codebase grows.

Common mistakes

Mistake Why it hurts Do this instead
Optimising jobs off the critical path No wall-clock improvement Measure the chain of dependent jobs first
Cache keyed only on branch Stale or useless caches Key on lockfiles, check hit rates
Splitting tests by file count Unbalanced shards Split by recorded timings
Affected-only builds without a full main build Missed breakages Full suite after merge
Retrying flaky tests forever Hidden cost and noise Quarantine with owners and deadlines

Key takeaways

  • Wall-clock time is the critical path; find it before optimising.
  • Remove unnecessary dependencies between jobs first; it is free.
  • Cache expensive, stable work with good keys, and measure hit rates.
  • Shard tests by timing, and build only what changed in monorepos.
  • Fix flakiness and right-size runners based on benchmarks.

References