CI was the slowest part of shipping. Here’s how we fixed it.

Every pull request has to wait for CI before it can merge, and every merge waits again before it deploys. When a check is too slow to sit through, people switch to other tasks. When it fails, they have to come back, fix it, push again and wait for the whole run again. We build our own search database and data warehouse in-house and make upgrades to them all the time. This means the team ships several hundred PRs a week. At that pace, CI is one of the largest limits on how fast we can iterate.

We spent three weeks measuring where the time and money went across our five main codebases, then cut it.

Where the time went.

We pulled every CI job from the last month out of the GitHub Actions API and ranked them by runner size times minutes. A handful of large jobs took most of the time and most of the money, and in each one we timed every step.

The worst offender was the test suite for our Search and DeepResearch infrastructure: about 3,300 tests run one file at a time on a 2-core runner, followed by a conformance check against our public SDKs. It was our slowest check. In eight days it timed out 14 times, and in 13 of those the tests had already passed.

What it adds up to.

A faster check matters in proportion to how often people wait for it. Multiplying each codebase's PR check runs per week by the median wait gives the total time the team spends waiting on CI each week. After the changes, that total is 54% lower.

What we changed.

About 50 changes, in five kinds. The cheapest wins came from asking what used each piece of work before we tried to make it faster.

What did not work.

Several obvious fixes actually made CI slower or more expensive, so every change ran in CI several times to validate them before it stayed. A 16-core runner for our largest Python suite would have doubled its cost to save seconds. Splitting our Ray tests into shards added 24% more runner minutes and did not shorten the wait. Moving our Rust tests from 8 cores to 4 cut compute by a quarter but made the check 51% slower, so it stayed on 8. A cached image builder was slower on every run we tried. Splitting work into shards helps the wait and can cost more money, because each shard repeats checkout, install and compile; we only kept a split when the total runner minutes made sense too.

How we work now.

The last 8 to 9% of wasted CI came from how we pushed changes. A push that a newer push replaced within minutes still paid for most of its run. Release PRs re-ran the full suite every time anything merged. Additionally, bots that bump image versions in our deploy config pushed to their PRs after every image build, even when the change was identical to the last one, and each push started a full CI run. The fix was an organisation-wide change in behaviour, not code: open a PR as a draft, run the local checks, push once per batch of commits, and mark it ready when it is done. This meant heavier jobs wait to be ready instead of repeatedly running CI.

Where this leaves us.

CI sets the pace for everything else we build, so we now treat it like any other part of the product: it has a cost, a latency and a failure rate, and we measure all three. Most of the gain came from deleting work that nothing used and from fixing the stalls and flakes that made people re-run checks. Bigger machines added little. The team now merges nearly twice as many PRs a week as it did in September while waiting only about half as long for CI. Going forward, every new check in the pipeline has to answer two questions before it ships: what reads its output coverage, and what it costs us to wait for it.