# Boat CI skills

Boat reads the GitHub Actions workflow and the public run history of a public GitHub repository, draws the pipeline, and estimates which changes would make it faster. This document is written for a coding agent working inside a checkout of such a repository. It is enough on its own: fetch the report, pick one skill, change the workflow, measure, repeat.

Boat is an independent project. It is not an official boat.dev product.

## What is in here

- 8 speedup practices. Boat checks each one against a repository and returns it in the report with an estimated saving and the formula behind it.
- 17 measured changes from one repository, pingdotgg/t3code: every performance change to its `.github/workflows/ci.yml`, with the median CI run time before and after. 7 were followed by a faster median, 5 by a slower one, 5 have no before measurement. The ones that did not help are listed too.
- A sharding procedure (section 5): record per-file test time, pack by duration, choose the number of shards, change the workflow matrix, verify. It includes a case of ours where more machines did not help.
- Sources today, three kinds. Measured by us: the pingdotgg/t3code workflow history above, plus test runs on boat sandboxes for pingdotgg/t3code and pandas-dev/pandas (apache/kafka: setup only, no test run), and one walk-through of section 5 on pingdotgg/t3code in local Docker. Reported by others: 12 published cases and posts, each with its author and link. Estimated: every saving in the report, and anything labelled so here, with its formula.
- Every number in this document is labelled `Measured`, `Reported by <who>`, `Estimated` or `Assumed`. Measured means it can be traced to a run record in the Boat source tree or to public GitHub Actions runs. Assumed means a constant inside an estimate that nobody measured; section 3 lists each one with a way to measure it. Keep the label when you quote a number. Commands are labelled `Verified` when we ran them as written and kept the record, and `Unverified` otherwise.
- Trying a plan automatically and optimizing continuously are planned. They are not available today.
- Boat does not run an agent for you. You, the agent reading this, do the work in the repository.

## 1. Get the CI report

Find the repository: `git remote get-url origin` gives `<owner>/<repo>`. Only public GitHub repositories that use GitHub Actions are supported.

Both calls below send one thing: the public `<owner>/<repo>` name. Boat then reads the public workflow files and the public run history from GitHub. No source code, token or secret leaves the checkout. Fetch the report before you edit anything: it holds the measured timings the practices are ranked by.

Read a saved report:

```
curl -s https://chaorders.com/boat/api/ci/<owner>/<repo>
```

If that returns 404 `{"error":"Report not found"}`, ask Boat to analyze the repository. The call is synchronous and can take up to about 45 seconds:

```
curl -s -X POST https://chaorders.com/boat/api/ci/analyze -H 'content-type: application/json' -d '{"repo":"<owner>/<repo>"}'
```

It returns `{ "status", "reportUrl", "report" }`; `report` is the same object the GET returns. Errors are JSON `{"error": "..."}`: 404 repository not found or not public, 422 no GitHub Actions workflow, 429 too many analyses (wait for the `retry-after` header, in seconds), 504 timed out (try again). A report is reused for 24 hours, so it will not show the effect of a change you just made.

A person can open the same report as a page: https://chaorders.com/boat/ci/<owner>/<repo>

## 2. Read the report

- `status`: use the report only when it is `ready`. `error` explains a failed one.
- `explanation`: one sentence on what the pipeline does and where the time goes.
- `summary`: the main workflow (`workflowPath`), the measurement window, `wallClock.p50Seconds` and `p95Seconds` (measured run time), `runsPerDay`, `rerunRate`, `failureRate`, `setupSecondsP50`. It is null when run history could not be fetched; then only the workflow structure is known.
- `jobs[]`: per job, measured `p50Seconds`, `setupSecondsP50` and `onCriticalPath`. The critical path decides the run time; work on it first.
- `pipeline.current`: jobs, `needs` edges, stages, matrix and shard counts as written in the workflow. `pipeline.cancelsStaleRuns`, `pipeline.hasPathFilter` and `pipeline.hasDependencyCache` say which practices are already in place.
- `suggestions[]`: one entry per practice below. `applies` says whether it would save anything here. `wallClockSavedSeconds` is time off one run, `runnerHoursSavedPerDay` is machine time. `evidence` is `estimated` unless stated otherwise; `formula` and `inputs` let you recompute every number.
- `projection`: estimated median run time if the critical-path suggestions are applied together.
- `recommendedMachines`: estimated number of parallel machines, with its formula.
- `changes[]`: earlier commits to the workflow with the measured median before and after, when there were enough runs.
- `limits[]`: what the numbers do not cover. Read it before quoting a number.

## 3. The skills

Every saving Boat reports for these practices is an estimate computed from the run history of the repository, not a measurement. The formula is listed so you can check it.

### Fork from a warm snapshot instead of repeating setup

- Report key: `warm-snapshot-fork`
- Why (the analyzer's reasoning, not a measurement): Each job on the critical path spends its first steps on checkout, tool setup and installs. A boat.dev sandbox forks from a snapshot that already holds that state, so the job starts at the first test step.
- Measured (boat/experiments/t3code/results.json, boat, one run per tier): forking took a median of 10.6 s over 4 machines and 10.1 s over 8 machines, so a forked job does not start at the first test step at once. We have no run that compares a forked job with the same job on a GitHub runner step by step.
- When it applies: The critical-path job spends a large part of its time before the first test step (`jobs[].setupSecondsP50` against `jobs[].p50Seconds`).
- What to change: The full saving needs a sandbox that forks from a warm snapshot, which a workflow edit cannot provide. Inside GitHub Actions, get part of it: cache dependency installs and tool downloads, use a prebuilt container image, and remove setup steps that a job does not use.
- Estimate (formula): `wallClockSavedSeconds = round1(max(0, criticalSetupP50Seconds - forkSeconds)); null when jobSampleRuns = 0`

### Balance shards by measured test time

- Report key: `balanced-shards`
- Why (the analyzer's reasoning, not a measurement): A sharded job finishes when its slowest shard does. Splitting tests by recorded duration brings the slowest shard down to the mean shard time.
- When it applies: A sharded matrix job has shards with clearly different durations (compare `jobs[].p50Seconds` across the shards of one job).
- What to change: Assign test files to shards by recorded duration instead of by count or name, using the timing-based split of the test runner or a checked-in timing file.
- Estimate (formula): `wallClockSavedSeconds = round1(max(0, slowestShardP50Seconds - meanShardP50Seconds)); null when shards < 2`

### Run the critical-path tests on 12 machines

- Report key: `more-shards`
- Why (the analyzer's reasoning, not a measurement): Sandboxes are billed per second, so the same test work spread over more machines costs about the same and finishes sooner. Only per-shard setup stays constant.
- Measured (boat/experiments/skill-validation/results.json, local Docker, not boat, median of 3 runs each): the same 5631 tests cost 75.6 s of machine time unsharded, 92.8 s on 2 shards and 124.7 s on 4. More shards finished sooner and cost more, not about the same: each shard pays its own start, imports and setup.
- When it applies: The critical-path job is dominated by test time, not setup, and the test runner can split its work.
- What to change: Add or raise a `strategy.matrix` shard dimension on the test job and pass the shard index and total to the test runner. Raise it in steps and measure each step: setup is paid once per shard.
- Estimate (formula): `T(N) = round1(setupSeconds + testTotalSeconds / N); tCurrent = T(currentShards); t8 = T(8); t12 = T(maxShards); wallClockSavedSeconds = round1(max(0, tCurrent - t12)); null when jobSampleRuns = 0`

### Cancel runs made stale by a newer push

- Report key: `cancel-stale-runs`
- Why (the analyzer's reasoning, not a measurement): Without concurrency cancellation, every push to a pull request keeps the previous run going to the end. Cancelling superseded runs frees that runner time.
- When it applies: `pipeline.cancelsStaleRuns` is false.
- What to change: Add a workflow-level `concurrency` block with a group keyed on the workflow and ref, and `cancel-in-progress` enabled for pull request runs. Keep default-branch runs uncancelled if deployments depend on them.
- Estimate (formula): `runnerHoursSavedPerDay = round3(assumedStaleRunFraction * runsPerDay * runnerSecondsPerRun / 3600); 0 when featurePresent = 1; null when workflowKnown = 0 or jobSampleRuns = 0`

### Skip jobs the change cannot affect

- Report key: `skip-unrelated-jobs`
- Why (the analyzer's reasoning, not a measurement): Path filters or a changed-files step let documentation-only and single-package changes skip jobs that cannot fail because of them.
- When it applies: `pipeline.hasPathFilter` is false and the repository has documentation or independent packages.
- What to change: Add `paths` or `paths-ignore` to the triggers, or a changed-files job whose outputs gate other jobs with `if:`. Prefer job-level `if:` for required checks, because a workflow skipped by a path filter leaves a required check pending.
- Estimate (formula): `runnerHoursSavedPerDay = round3(assumedSkippableFraction * runsPerDay * runnerSecondsPerRun / 3600); 0 when featurePresent = 1; null when workflowKnown = 0 or jobSampleRuns = 0`

### Cache dependencies between runs

- Report key: `dependency-cache`
- Why (the analyzer's reasoning, not a measurement): The workflow reinstalls dependencies on every job. Restoring them from a cache removes part of the setup time of every job.
- When it applies: `pipeline.hasDependencyCache` is false.
- What to change: Turn on the `cache` input of the language setup action in use, or add `actions/cache` keyed on the hash of the lockfile.
- Estimate (formula): `runnerHoursSavedPerDay = round3(assumedCacheableSetupFraction * runsPerDay * setupRunnerSecondsPerRun / 3600); 0 when featurePresent = 1; null when workflowKnown = 0 or jobSampleRuns = 0`

### Retry failed tests in parallel inside the same run

- Report key: `parallel-flaky-retry`
- Why (the analyzer's reasoning, not a measurement): A rerun repeats the whole workflow after someone notices the failure. Retrying only the failed tests on a fresh fork while the run is still going removes that second cycle.
- When it applies: `summary.rerunRate` is above zero.
- What to change: Retry only the failed tests inside the same run with the retry option of the test runner, and list the tests that needed a retry so they can be fixed. Do not use retries to hide a real failure.
- Estimate (formula): `wallClockSavedSeconds = round1(max(0, rerunRate * (assumedRerunWaitSeconds + rerunDurationSeconds))), expected per run`

### Test merges speculatively before they land

- Report key: `speculative-merge-queue`
- Why (the analyzer's reasoning, not a measurement): A red default branch costs at least one more full CI cycle to fix or revert. Testing each merge result on a fork before it lands keeps those failures off the default branch.
- When it applies: Push runs on the default branch fail often enough to matter (`summary.failureRate`, and the inputs of this suggestion).
- What to change: Add the `merge_group` trigger to the workflow. Turning on the merge queue is a repository setting: propose it to the maintainer instead of changing it.
- Estimate (formula): `defaultBranchFailureRate = defaultBranchFailures / defaultBranchRuns; wallClockSavedSeconds = round1(max(0, defaultBranchFailureRate * wallClockP50Seconds)), expected per merge; null when defaultBranchRuns < minDefaultBranchRuns`

Assumptions inside these estimates, as the report states them in `limits[]`. Where we hold a measurement, it follows the line:

- Assumed: Warm snapshot fork time is a constant of 5 s in the analyzer.
  - Measured (boat/experiments/t3code/results.json, boat, one run per tier): 10.6 s median over 4 parallel forks, 10.1 s median over 8 parallel forks. Measured (boat/experiments/pandas/results.json, boat, one value per run): 4.9 s, 5.3 s, 5.6 s, 7.4 s. The analyzer still uses the assumed value today, so its `warm-snapshot-fork` saving is optimistic by a few seconds.
- Assumed: Shard count is capped at 12 because the boat $20 plan starts at most 12 new machines per minute.
  - The plan limit in that sentence is the analyzer's statement. We hold no run record or published source for it, so treat the cap as a setting of the analyzer.
- Assumed: More-shards estimates assume test work splits evenly across shards, including jobs that are not sharded today.
- Assumed: Stale-run savings assume 15% of runner time goes to superseded runs; this fraction is an assumption, not measured.
- Assumed: Skip-unrelated-jobs savings assume 20% of runner time is spent on jobs the change cannot affect; this fraction is an assumption, not measured.
- Assumed: Dependency-cache savings assume 50% of setup time is cacheable; this fraction is an assumption, not measured.
- Assumed: The wait before a manual rerun is not measurable from run history and is assumed to be 0 s, so the flaky-retry saving is a lower bound.
- Assumed: The merge-queue saving assumes each default-branch failure costs exactly one more full CI cycle.

### Assumed values, and how to measure them in your repository

7 values inside the estimates were chosen, not measured: 5 constants and 2 simplifications. Each is listed with what it stands for and how to replace it with a number from the repository you are working in. A saving that uses an assumed constant is linear in it, so you can rescale the reported saving without asking Boat again. The commands read public run history and the local checkout only.

- Verified (boat/experiments/skill-verify/assumed-commands.json, 2026-11-20): the 3 commands below ran as written, with only the placeholders filled in, in a checkout of pingdotgg/t3code; each exited 0 and printed a result. The record keeps the filled-in command and its output. The values they printed describe that repository on that day and are not quoted here.

- Assumed: 15% for `cancel-stale-runs` (the share of runner time spent on runs that a newer push to the same branch had already replaced).
  - For the last 200 pull request runs, add up the seconds each run kept going after the next run on the same branch had started, and divide by the seconds of all runs.
  - This counts run wall clock, not runner time, and it groups by branch name, so two forks that use the same branch name are counted as one branch and push the number up. Read it as an upper bound.

  ```
  gh run list --workflow <file> --event pull_request --limit 200 --json headBranch,createdAt,updatedAt --jq '[group_by(.headBranch)[] | sort_by(.createdAt) | . as $r | range(0; length) | {s: (($r[.].updatedAt|fromdate) - ($r[.].createdAt|fromdate)), stale: (if . + 1 < ($r|length) then ([0, ($r[.].updatedAt|fromdate) - ($r[. + 1].createdAt|fromdate)] | max) else 0 end)}] | {runs: length, runSeconds: (map(.s)|add), staleSeconds: (map(.stale)|add)} | .staleFraction = (.staleSeconds / .runSeconds)'
  ```

  - Recompute: `runnerHoursSavedPerDay × staleFraction / assumedStaleRunFraction`
- Assumed: 20% for `skip-unrelated-jobs` (the share of runner time spent on jobs that the change could not affect).
  - In the checkout, count how many of the last 200 commits on the default branch touch only paths that no job needs. Set `skip` to the paths you intend to filter; the example counts Markdown files and `docs/`.
  - This is the share of runs that could skip everything, which is a lower bound: it does not count runs where only some jobs could be skipped.

  ```
  git log -200 --first-parent --name-only --format='@%H' origin/<default-branch> | awk -v skip='(\\.md$|^docs/)' '/^@/ { if (n) { total++; if (other == 0) skippable++ } n = 0; other = 0; next } NF { n++; if ($0 !~ skip) other++ } END { if (n) { total++; if (other == 0) skippable++ } printf "%d of %d commits touch only skippable paths (%.1f%%)\n", skippable, total, 100 * skippable / total }'
  ```

  - Recompute: `runnerHoursSavedPerDay × skippableShare / assumedSkippableFraction`
- Assumed: 50% for `dependency-cache` (the share of setup time that a dependency cache would remove).
  - List the step durations of one successful run. In the critical-path job, add the steps that download or install dependencies and tools, and divide by the seconds of all steps before the first test step.
  - That is the most a cache can remove. To measure what it does remove, add the cache in a pull request and run the same command on a run with a cache hit.

  ```
  gh run view <run-id> --json jobs --jq '.jobs[] | select(.conclusion=="success") | {job: .name, steps: [.steps[] | {name, seconds: ((.completedAt|fromdate) - (.startedAt|fromdate))}]}'
  ```

  - Recompute: `runnerHoursSavedPerDay × cacheableShare / assumedCacheableSetupFraction`
- Assumed: 5 s for `warm-snapshot-fork` (the time to start a machine from a warm snapshot until it runs its first command).
  - Measured (boat/experiments/t3code/results.json, boat, one run per tier, from the fork call to the first successful command, CLI round trips included): 10.6 s median over 4 parallel forks, 10.1 s median over 8 parallel forks. Measured (boat/experiments/pandas/results.json, boat, one value per run): 4.9 s, 5.3 s, 5.6 s, 7.4 s. The assumed value is at the low end of what we measured, so the `warm-snapshot-fork` saving and `projection` are optimistic by a few seconds.
  - On GitHub Actions there is no fork, so this saving cannot be reached by a workflow edit. If you run tests on your own sandboxes, time from the start request to the first command returning, in parallel for as many machines as you will use, and take the median.
  - Recompute: `max(0, criticalSetupP50Seconds - yourForkSeconds)`
- Assumed: 0 s for `parallel-flaky-retry` (the time a person waits before pressing rerun on a failed run).
  - Run history does not show it. The saving is a lower bound; no command replaces this value.
- Assumed: test work splits evenly across shards (`more-shards`). Replace it with the planned load of the heaviest shard from the packer in section 5, step 3 (`node ci/shard.mjs loads <N> ci/test-timings.json`).
- Assumed: each default-branch failure costs exactly one more full CI cycle (`speculative-merge-queue`). No command replaces this value here.

## 4. Measured changes from pingdotgg/t3code

Source: public GitHub Actions data for pingdotgg/t3code, `.github/workflows/ci.yml`, 100597 runs, data generated 2026-11-20. Each number is measured: the median run time of main-branch CI in the 7 days before and after the change. These are observations, not controlled experiments: the repository and its test suite kept growing in the same period, so a change can be followed by a slower median. Use them as examples of what a real team changed and in what order, and read the slower ones as warnings: a technique that helped one pipeline can cost another.

Case page with charts: https://chaorders.com/boat/ci/t3code

### Faster machines (2)

- 2026-02-27 · Move to Blacksmith runners. Measured: before 2 min 01 s (11 runs), after 1 min 05 s (70 runs). Result: faster by 56 s. https://github.com/pingdotgg/t3code/pull/111
- 2026-10-05 · Move the transfer report job to Blacksmith. Measured: before 2 min 14 s (244 runs), after 2 min 54 s (147 runs). Result: slower by 40 s. https://github.com/pingdotgg/t3code/pull/16178

### Split jobs (1)

- 2026-06-04 · Split the single job into parallel jobs. Measured: before 3 min 57 s (23 runs), after 2 min 14 s (61 runs). Result: faster by 103 s. https://github.com/pingdotgg/t3code/pull/2940

### Shard tests (4)

- 2026-08-21 · Parallelize tests, split out Rust checks. Measured: before 3 min 36 s (198 runs), after 1 min 30 s (121 runs). Result: faster by 125 s. https://github.com/pingdotgg/t3code/pull/7286
- 2026-10-01 · Run PR checks side by side, balance server shards. Measured: before 2 min 09 s (169 runs), after 2 min 26 s (352 runs). Result: slower by 17 s. https://github.com/pingdotgg/t3code/pull/14025
- 2026-11-11 · Split web tests over two runners. Measured: before 4 min 04 s (752 runs), after 2 min 57 s (2007 runs). Result: faster by 67 s. https://github.com/pingdotgg/t3code/pull/34554
- 2026-11-12 · Run server test files in parallel on each shard. Measured: before 3 min 53 s (1169 runs), after 2 min 54 s (1783 runs). Result: faster by 59 s. https://github.com/pingdotgg/t3code/pull/35618

### Cache (4)

- 2026-10-15 · Cache Chromium for server test shards. Measured: before: not measured, after 3 min 31 s (162 runs). Result: not comparable. https://github.com/pingdotgg/t3code/pull/19632
- 2026-10-15 · Cache pnpm packages on sticky disks. Measured: before: not measured, after 3 min 31 s (162 runs). Result: not comparable. https://github.com/pingdotgg/t3code/pull/19634
- 2026-10-25 · Cache server tests individually across shards. Measured: before 3 min 02 s (356 runs), after 3 min 38 s (1093 runs). Result: slower by 36 s. https://github.com/pingdotgg/t3code/pull/24966
- 2026-10-28 · Shards survive a throttled task cache. Measured: before 3 min 30 s (680 runs), after 3 min 43 s (1163 runs). Result: slower by 13 s. https://github.com/pingdotgg/t3code/pull/26332

### Skip work (3)

- 2026-07-30 · Cancel stale runs, drop redundant setup. Measured: before 3 min 18 s (126 runs), after 3 min 37 s (90 runs). Result: slower by 20 s. https://github.com/pingdotgg/t3code/pull/4802
- 2026-08-21 · Boot the macOS lint runner only for native changes. Measured: before 3 min 35 s (199 runs), after 1 min 30 s (120 runs). Result: faster by 125 s. https://github.com/pingdotgg/t3code/pull/7283
- 2026-10-15 · Skip jobs a PR cannot affect. Measured: before: not measured, after 3 min 31 s (162 runs). Result: not comparable. https://github.com/pingdotgg/t3code/pull/19587

### Shorten critical path (3)

- 2026-10-15 · Run typecheck inside the Build job. Measured: before: not measured, after 3 min 31 s (162 runs). Result: not comparable. https://github.com/pingdotgg/t3code/pull/19633
- 2026-10-15 · Cut per-job setup time. Measured: before: not measured, after 3 min 31 s (162 runs). Result: not comparable. https://github.com/pingdotgg/t3code/pull/20044
- 2026-11-06 · Boot the built web client in Build, not Test Web. Measured: before 3 min 46 s (1165 runs), after 3 min 45 s (1234 runs). Result: faster by 1 s. https://github.com/pingdotgg/t3code/pull/30752

## 5. Find the fastest descent by sharding

The fastest plan is the one where no single piece is left to shorten: every shard on the critical path finishes at about the same time, and that time is close to the next-slowest job. Follow the steps in order. Each step names the evidence behind it; file paths are run records in the Boat source tree, which is not public yet.

### Step 1. Find the job and the step to split

From the report, take the job with `onCriticalPath: true` and the largest `p50Seconds`. Subtract `setupSecondsP50`: if setup is most of the job, sharding multiplies setup and saves little, so fix setup first (`dependency-cache`, `warm-snapshot-fork`). Then open one recent run of that job (`gh run view <run-id> --json jobs`) and find the slowest test step by its `startedAt` and `completedAt`. Get a run id with `gh run list --workflow <summary.workflowPath> --status success --limit 5 --json databaseId` (Verified, boat/experiments/skill-verify/assumed-commands.json, 2026-11-20: ran with the workflow file filled in and returned run ids), and pick a run in which that job ran: a skipped job has no step times. If the job name ends in a shard number, such as `Test Web 2`, the suite is already split and the job to change is the matrix job behind that name. Compare the `p50Seconds` of its shards: if they are close, the split is already balanced and the only thing left to change is N.

- Measured (boat/experiments/t3code/results.json, one run on one large boat machine, every step in a row): the whole of pingdotgg/t3code CI took 1117.4 s; the slowest step was `test_web` at 469.1 s. That step is the one worth splitting first.

### Step 2. Record how long each test file takes

Run the suite once, unsharded, and save the seconds per unit as a JSON object `{ "<unit>": <seconds> }` in `ci/test-timings.json` (`mkdir -p ci` first). The key must be spelled exactly as the list command prints the unit, or the packer treats every unit as unknown. For each runner below: the command that records times, the command that converts its output, the command that lists units, and the command that runs one shard from `shard-files.txt`.

**Vitest** (unit: test file)

```
npx vitest run --reporter=default --reporter=json --outputFile=vitest.json
node -e 'const path=require("path");const t={};for(const f of require("./vitest.json").testResults)t[path.relative(process.cwd(),f.name)]=(f.endTime-f.startTime)/1000;console.log(JSON.stringify(t,null,1))' > ci/test-timings.json
git ls-files "*.test.ts" | node ci/shard.mjs <index> <N> ci/test-timings.json > shard-files.txt
xargs npx vitest run < shard-files.txt
```

- Verified (boat/experiments/skill-verify/results.json, 2026-11-20): all four commands ran as written with Vitest (version output: vitest/5.1.1 linux-x64 node-v26.13.1) on a fixture project of 4 test files that sleep 400, 200, 100, 50 ms. The timing keys equalled the listed units, the slowest unit was the one with the longest sleep, 2 shards covered every unit exactly once, and every shard exited 0. A fixture is not your repository: check the first two commands on one real run before relying on them.

**Jest** (unit: test file)

```
npx jest --json --outputFile=jest.json
node -e 'const path=require("path");const t={};for(const f of require("./jest.json").testResults)t[path.relative(process.cwd(),f.name)]=(f.endTime-f.startTime)/1000;console.log(JSON.stringify(t,null,1))' > ci/test-timings.json
git ls-files "*.test.js" | node ci/shard.mjs <index> <N> ci/test-timings.json > shard-files.txt
xargs npx jest --runTestsByPath < shard-files.txt
```

- Verified (boat/experiments/skill-verify/results.json, 2026-11-20): all four commands ran as written with Jest (version output: 30.5.2) on a fixture project of 4 test files that sleep 400, 200, 100, 50 ms. The timing keys equalled the listed units, the slowest unit was the one with the longest sleep, 2 shards covered every unit exactly once, and every shard exited 0. A fixture is not your repository: check the first two commands on one real run before relying on them.

**pytest** (unit: test file)

```
pytest -o junit_family=xunit1 --junitxml=report.xml
python3 -c "import json,collections,xml.etree.ElementTree as E; t=collections.Counter(); [t.update({c.get('file'): float(c.get('time'))}) for c in E.parse('report.xml').iter('testcase')]; print(json.dumps(t, indent=1))" > ci/test-timings.json
git ls-files "test_*.py" "*/test_*.py" | node ci/shard.mjs <index> <N> ci/test-timings.json > shard-files.txt
xargs pytest < shard-files.txt
```

- Verified (boat/experiments/skill-verify/results.json, 2026-11-20): all four commands ran as written with pytest (version output: pytest 9.2.0) on a fixture project of 4 test files that sleep 400, 200, 100, 50 ms. The timing keys equalled the listed units, the slowest unit was the one with the longest sleep, 2 shards covered every unit exactly once, and every shard exited 0. A fixture is not your repository: check the first two commands on one real run before relying on them.

**go test** (unit: package)

```
go test -count=1 -json ./... > go-test.json
jq -s '[.[] | select(.Test == null and (.Action == "pass" or .Action == "fail")) | {(.Package): .Elapsed}] | add' go-test.json > ci/test-timings.json
go list ./... | node ci/shard.mjs <index> <N> ci/test-timings.json > shard-files.txt
xargs go test -count=1 < shard-files.txt
```

- Verified (boat/experiments/skill-verify/results.json, 2026-11-20): all four commands ran as written with go test (version output: go version go1.26.3 linux/amd64) on a fixture project of 4 packages that sleep 400, 200, 100, 50 ms. The timing keys equalled the listed units, the slowest unit was the one with the longest sleep, 2 shards covered every unit exactly once, and every shard exited 0. A fixture is not your repository: check the first two commands on one real run before relying on them.

**Gradle (JUnit)** (unit: test class)

```
gradle test
python3 -c "import json,glob,xml.etree.ElementTree as E; print(json.dumps({s.get('name'): float(s.get('time')) for s in (E.parse(f).getroot() for f in glob.glob('**/build/test-results/test/TEST-*.xml', recursive=True))}, indent=1))" > ci/test-timings.json
jq -r 'keys[]' ci/test-timings.json | node ci/shard.mjs <index> <N> ci/test-timings.json > shard-files.txt
gradle test $(sed 's/^/--tests /' shard-files.txt)
```

- Verified (boat/experiments/skill-verify/results.json, 2026-11-20): all four commands ran as written with Gradle (JUnit) (version output: Gradle 9.9.1) on a fixture project of 4 test classes that sleep 400, 200, 100, 50 ms. The timing keys equalled the listed units, the slowest unit was the one with the longest sleep, 2 shards covered every unit exactly once, and every shard exited 0. A fixture is not your repository: check the first two commands on one real run before relying on them.

Notes on the commands above. Vitest and Jest report absolute paths, so the converter makes them relative to the directory you run it in; run it at the repository root. pytest needs `-o junit_family=xunit1`: the default report has no `file` attribute, and the class name alone does not map back to a file when tests live in classes. `go test` needs `-count=1`, because a cached package reports no real time. The Gradle list comes from the timings file, so a new test class is not in any shard until timings are refreshed: run the unsharded suite on the default branch on a schedule to catch them. In a monorepo, run the collection inside the package whose test job you are splitting, with the test command that job uses.

- Unverified: Playwright (`playwright test --reporter=json`) and other runners were not run by us; the same four-command shape should apply, but check each command yourself.

Check the timings before you pack with them. Four checks, each from the walk-through recorded in `boat/experiments/skill-validation/friction.md`:

1. The run that produced them exited 0. A run that crashes part way can still write a report that lists every file as passed, with zero duration for the files it never reached; those files would get a weight of 0.
2. The timings file has as many units as the list command prints (`jq length ci/test-timings.json` against `<list command> | wc -l`). The packer prints both counts on stderr: `386 files, 0 without a recorded time`. If the second number is not 0, the keys do not match the list or the list is wrong. A list pattern that looks right can still select the wrong set: files of another test project, or a missed extension such as `.test.tsx`.
3. If the repository wraps the runner (a task runner, a script with a cache), run the runner itself with the flags the job uses, so the cache cannot skip the work, and keep those flags in the command that runs one shard.
4. The recorded seconds account for the run. Add up the recorded seconds and compare with `cores × wall clock` of the same run. If the sum is far smaller, the runner spends most of its time on work it does not attribute to a file in the report: importing, transforming, starting workers. Packing by the recorded seconds alone then balances the small part and ignores the large one. Give every unit a constant on top: `per_file = (cores × wall_clock − recorded_sum) / files`, and pass it to the packer as a fourth argument (`node ci/shard.mjs <index> <N> ci/test-timings.json <per_file>`).

- Measured (boat/experiments/skill-validation/results.json, pingdotgg/t3code at 338352d, local Docker, one 4-CPU container per shard, not GitHub runners and not boat machines; task: job Test Web, step Test, Vitest project unit only (the browser project lost its browser connection in both attempts and was dropped)): Vitest recorded 26.4 s over 386 test files, while the unsharded run took a median of 74.4 s on 4 cores. The recorded seconds are 9% of the core seconds. Estimated (formula above, assumes all 4 cores were busy for the whole run): 0.70 s per file.
- Measured (boat/experiments/skill-validation/friction.md, same checkout): the first attempt to collect timings ran the browser tests too, lost its browser connection and exited 1, yet its report said `"success": true` and listed all 634 files as passed, 82 of them with no assertions and no duration.

- Measured (boat/experiments/pandas/timings.json, written by boat/experiments/fanout/bench.py from JUnit XML): 915 test files of pandas-dev/pandas sum to 294.7 s; the heaviest, `pandas/tests/extension/test_arrow.py`, takes 27.0 s.

### Step 3. Pack by duration, then choose N

Sort files from longest to shortest and give each one to the shard with the least time so far. Splitting by file count or by name hash puts the same number of files on every shard, but file durations are far from equal, so one shard ends up with the long files and every other machine waits for it. The run ends when the slowest shard ends.

- Measured (boat/experiments/pandas/results.json, one run each, 8 boat machines, 966 test files of pandas-dev/pandas, same commit): in the first run the shards ran 16.5 s to 82.4 s, wall clock 84.2 s (259,603 tests). In the second the shards ran 22.4 s to 31.6 s, wall clock 32.7 s (259,601 tests). Same commit and 8 machines of the same size; the shard layout differed. What made the first layout worse is not established: the record of each run gives its packing as "measured seconds per test file from an earlier run" for both, while a note in the same file says the first was balanced by file size or by partial timings and the second by the seconds per file measured in the first. What the two runs do show is the cost of an uneven layout: the run ends with its slowest shard. 26 tests failed in both runs, so the result is not a green run.
- Measured (boat/experiments/pandas/results.json, one run each, a subset: 4 test directories: frame, series, indexes, groupby (432 files), 77286 tests): 1 boat machine 86.6 s, 8 machines 17.2 s plus 5.3 s to fork them: 5.03 times faster, 4.06 times with the fork counted. Packed by test file size in bytes, the shards still ran 6.3 s to 16.3 s: file size is a weak stand-in for duration. 1 test failed on 1 machine and on 8 alike. This compares boat with boat on a subset and says nothing about the whole pandas CI or about GitHub runners.

Choosing N:

```
wall_clock(N) ≈ max(start + setup + heaviest_shard(N), next_slowest_job)
heaviest_shard(N) ≥ max(total_test_seconds / N, heaviest_single_file)
```

`start + setup` is paid once per shard: on GitHub Actions it is `jobs[].setupSecondsP50`; on forked sandboxes it is fork time plus checkout. Raise N until one of three things happens, then stop: the heaviest shard is one file (split that file or stop), the next-slowest job is now the critical path (work on that job instead), or one more machine saves less than 5% of wall clock. The planned loads can only rule a value of N out. They cannot confirm one: apply the 5% rule to measured runs of N and N+1, not to the plan. And do not apply the one-file rule until check 4 of step 2 passes, because a file that looks like the whole shard in the recorded seconds may be a small part of it in the run.

- Estimated (formula: greedy longest-first packing of the per-file seconds in boat/experiments/pandas/timings.json; assumes a file takes the same time on any shard and ignores start and setup): heaviest shard for N=2: 147.4 s by duration against 147.7 s by count in name order; N=4: 73.7 s by duration against 91.0 s by count in name order; N=8: 36.8 s by duration against 55.9 s by count in name order; N=16: 27.0 s by duration against 36.3 s by count in name order. From N=11 on, the heaviest file (27.0 s) is the floor and more machines buy nothing.
- Measured (boat/experiments/t3code/results.json): the stop rule used in our own runs is "smallest N after which one more machine saves less than 5% of wall clock (max 12)".

- Measured (boat/experiments/skill-validation/results.json, pingdotgg/t3code at 338352d, local Docker, one 4-CPU container per shard, not GitHub runners and not boat machines; 3 runs each, alternating, 5631 tests in 386 files on every run, 0 failed): an agent that had only this document packed by the recorded seconds alone, as the document then said. The one-file rule told it to stop at N=2, because `src/reactCompilerCoverage.test.ts` (13.9 s recorded) was a shard by itself. Unsharded median 74.4 s (74.4 s to 75.1 s); 2 shards 73.6 s (71.2 s to 74.4 s), not faster: the shard with the one heavy file finished in about 16.6 s and the shard with the other 385 files took as long as the whole suite. 4 shards 34.8 s (34.3 s to 35.9 s), 2.14 times faster, a value the plan said would gain nothing.
- Measured (same place, round 2, run by us after the fix, 3 runs each, alternating; the only change is 0.7 passed to the packer as seconds per file): unsharded median 75.6 s (75.1 s to 75.7 s); 2 shards 47.5 s (47.1 s to 48.3 s), 1.59 times faster, shards 43.8 s to 48.3 s; 4 shards 37.2 s (36.8 s to 38.0 s), 2.03 times faster, shards 28.0 s to 38.0 s. 5631 tests on every run, 0 failed. Machine seconds went from 75.6 s unsharded to 92.8 s on 2 shards and 124.7 s on 4. The constant fixed 2 shards and did not improve 4: round 1 had reached 34.8 s on 4 shards without it. In round 2 the shard holding the heavy file was the slowest of the 4 on every run (36.8 s to 38.0 s, the other three 28.0 s to 30.0 s), so one constant per file is a rough correction. After the first sharded run, read the real duration of each shard and move files off the slowest one.

A packer with no dependencies, `ci/shard.mjs`. It reads the file list on stdin and prints the files of one shard; files with no recorded time get the median, and the optional fourth argument is added to every file:

```js
import fs from 'node:fs'
const [index, total, timingsPath, perFile = '0'] = process.argv.slice(2)
const timings = JSON.parse(fs.readFileSync(timingsPath, 'utf8'))
const files = fs.readFileSync(0, 'utf8').split('\n').filter(Boolean)
const known = files.map((f) => timings[f]).filter((s) => s !== undefined).sort((a, b) => a - b)
const fallback = known.length ? known[Math.floor(known.length / 2)] : 1
const weight = (f) => (timings[f] ?? fallback) + Number(perFile)
const bins = Array.from({ length: Number(total) }, () => ({ load: 0, files: [] }))
for (const f of [...files].sort((a, b) => weight(b) - weight(a) || (a < b ? -1 : 1))) {
  const bin = bins.reduce((min, b) => (b.load < min.load ? b : min))
  bin.files.push(f)
  bin.load += weight(f)
}
console.error(files.length + ' files, ' + (files.length - known.length) + ' without a recorded time')
if (index === 'loads') console.log(bins.map((b) => b.load.toFixed(1)).join(' '))
else console.log(bins[Number(index)].files.join('\n'))
```

`git ls-files "*.test.ts" | node ci/shard.mjs loads 4 ci/test-timings.json` prints the planned seconds per shard, so you can compare values of N before running anything.

### Step 4. The smallest workflow change

```yaml
jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        shard: [0, 1, 2, 3]
    steps:
      - uses: actions/checkout@v4
      - run: git ls-files "*.test.ts" | node ci/shard.mjs ${{ matrix.shard }} 4 ci/test-timings.json > shard-files.txt
      - run: xargs npx vitest run < shard-files.txt
```

Keep the existing setup steps of the job; replace only the test command, using the list and run commands of your runner from step 2. Check `ci/test-timings.json` in. Refresh it from a default-branch run on a schedule (weekly is a reasonable start) and whenever the shard spread grows; a new file without a timing gets the median until then. If the suite is already split with a runner flag such as `--shard=<i>/<N>` (Vitest, Jest, Playwright), that split is by file, not by duration: compare the shard durations in the report before replacing it. If they are already close, keep the flag and change only the matrix list. Keep every flag of the original test command after `xargs` (project, config, `--passWithNoTests`). Vitest treats a file argument as a substring filter, so `src/a.test.ts` also selects `src/a.test.tsx`; the test count in step 5 is the guard.

### Step 5. Prove it, or revert

Run the old and the new workflow on the same commit, at least 3 successful runs each, alternating old and new so that load on shared runners falls on both, and compare medians of the whole run, not of the test job. Check three things: every new run is faster than every old run (a lower median with overlapping ranges is noise, not a gain), the same number of tests ran (compare the test counts in the logs; a shard that silently runs nothing looks fast), and runner minutes, the sum of the durations of all shards, did not grow beyond what the person you work for accepts. If the new runs are not faster, revert and write down why.

When sharding does not help. Our own counterexample:

- Measured (boat/experiments/t3code/results.json, pingdotgg/t3code at 338352dee2f4, one run per tier): default boat machines forked from one warm machine ran the CI in 484.4 s on 4 machines and 402.4 s on 8. Measured (GitHub Actions CI runs of pingdotgg/t3code, pull requests, 2026-10-20 to 2026-11-20): the project's own CI has a p50 of 202 s and a p90 of 279 s on GitHub Actions. Eight boat machines were 2.0 times slower than the GitHub median. Sharding more did not fix it.
- Measured (boat/experiments/t3code/runs.jsonl, run boat-8x-default): the reason is one task that was not split. The packages `test` task ran 351.8 s on one machine while the server tests were spread over 16 shards. Doubling the machines from 4 to 8 took only 82.0 s off: the run cannot end before that task does, whatever the other machines do. Forking took a median of 10.1 s over 8 machines.
- Neither run is a clean pass: all 25 of 25 CI tasks exited 0 on 8 machines and 16 of 16 on 4, but the dispatcher marked the `sync` step (checkout and offline install) failed on every machine, and the cause is not confirmed.

So do not expect a gain from sharding when: the job is not on the critical path; setup is most of the job; one unsplit task or one file is longer than the balanced shard; the tests share a database, port or fixture that cannot run in parallel; or the new machines are slower than the ones you leave.

## 6. Measured runs on boat machines

These are our own runs of two open-source test suites on boat sandboxes, with the run records kept in the Boat source tree. Every wall-clock number is one run, not a median; fork times are the median over the machines of one run, and are marked as such. They are evidence for the method in section 5, not a claim about price or about speed against GitHub: the one direct comparison we have is slower than GitHub.

### pingdotgg/t3code (boat/experiments/t3code/results.json)

- Measured, boat: 1 large machine, every step in a row, 1117.4 s; failed steps: test_web, test_server (one test each, both fail outside GitHub Actions).
- Measured, boat: 4 default machines 484.4 s (fork 10.6 s); 8 default machines 402.4 s (fork 10.1 s). The `sync` step was marked failed on every machine in both.
- Measured, GitHub Actions (GitHub Actions CI runs of pingdotgg/t3code, pull requests, 2026-10-20 to 2026-11-20): p50 202 s, p90 279 s.
- Measured, local Docker, not boat: containers limited to the same size, run alone, 1 container 1195.9 s, 8 containers 353.2 s; starting a container from the warm image took a median of 1.2 s. These are not boat numbers. Neither local run passed every task.
- Measured, local Docker, not boat: when four tiers ran at the same time on the same hosts, 1 container took 1321.4 s and 8 took 410.4 s, against 1195.9 s and 353.2 s alone.

### pandas-dev/pandas (boat/experiments/pandas/results.json)

- Measured, boat: setup once on the base machine 165.9 s, not part of any test wall clock.
- Measured, boat, subset (4 test directories: frame, series, indexes, groupby (432 files)): 1 machine 86.6 s, 8 machines 17.2 s, fork 5.3 s.
- Measured, boat, 966 test files on 8 machines: 84.2 s in one run and 32.7 s in the next, with a different shard layout (section 5, step 3; the cause of the difference is not established). 26 tests failed in both.
- Not measured: the same 966 files on 1 machine. That run was killed 2 times before finishing, so there is no whole-suite speedup figure.

### apache/kafka (boat/experiments/kafka/results.json)

- Measured, boat: setup 132.5 s, of which compiling the test classes of one module 76.3 s. Status `not-run`: 0 test runs, so there is no test wall clock and no speedup for kafka.
- Measured, GitHub Actions (20 GitHub Actions runs of apache/kafka read on 2026-11-20): the job "build / JUnit tests Java 25" has a p50 of 7805.5 s. It is listed as the size of the target, not as a comparison.

### What went wrong, as steps

These apply when you run a test suite on forked sandboxes instead of GitHub runners. Commands are the ones in our scripts.

1. Drop ambient capabilities before running tests. Processes on boat machines carry the ambient capability `CAP_DAC_READ_SEARCH`, so tests that expect a permission error pass or fail differently. Start the test command with `setpriv --ambient-caps -all --inh-caps -all <command>` (boat/experiments/t3code/ci-steps.sh). Check with `grep CapAmb /proc/self/status`.
2. Under zero data retention a named snapshot cannot be saved. Keep one machine with dependencies installed running, and fork it in parallel: `boat fork <machine-id> --no-env --no-snapshots --ttl 3600 --json`, once per shard (boat/experiments/t3code/shard.mjs).
3. `boat exec <id> --json -- <command>` prints one JSON frame per line for a long command. Join the `data` of the `stdout` frames and read `exitCode` from the last frame; do not parse the output as one JSON document (boat/experiments/t3code/shard.mjs).
4. Measure the baseline alone. Tiers that run at the same time on shared hosts slow each other down (the two local Docker lines above), and a slow baseline makes every speedup look larger.
5. After sharding one suite, look for the next unsplit task. In our 8-machine run it was the packages `test` task at 351.8 s (section 5, step 5).
6. Count tests on both sides. A fan-out that reports a different number of tests than the single-machine run has lost or duplicated files.

## 7. Reported by others

12 results and practices that other teams published. None of these numbers were measured by Boat. Each line says who reported it, where, and what the number covers. Use them to choose what to try, and quote them only with the name attached.

- Reported by Owen Gretzinger, Boardy (https://x.com/owengretzinger/status/2116291924219060319): pull request CI went from up to 40 minutes to about 2 minutes, and getting a runner from up to 11 minutes to 8 seconds, after faster runners, fixed cold pull request caches, removed repeated setup, parallel jobs, and affected-project selection with a nightly full suite. Monthly CI spend rose from USD 230 to USD 4,300 at a changed workload. Scope: author-reported in a first-hand article on X; worst case before, typical after; not measured by Boat. How to use it: Sharding was one of several changes and the bill grew. Measure queue time separately from run time, and report runner minutes next to wall clock.
- Reported by DOSS engineering (https://careers.doss.com/resources/eng-blog/how-we-halved-our-deploy-times-mostly-by-doing-less): duration-balanced shards took one test phase from 10 to about 7 minutes; with other changes, median production deploy time fell from 28.7 to about 14 minutes (111 successful runs in the baseline). Scope: first-hand engineering article; not measured by Boat. How to use it: The closest outside case for section 5: same shard count, shorter phase, only from balancing by duration.
- Reported by Linear engineering (https://linear.app/now/ci-bottleneck-reworked): jobs ran 34% faster on average across the two days on either side of a runner switch; they also removed repeated setup, moved optional cache writes off the gate, and tuned job and test parallelism. Scope: first-hand engineering article, two days on each side; not measured by Boat. How to use it: Machine speed and repeated setup are separate levers from sharding. Check them when the report shows setup dominating the critical-path job.
- Reported by the author of humansystems.dudzik.co (https://humansystems.dudzik.co/p/when-agents-make-ci-the-bottleneck): full pull request suite wall time went from about 21 to 13 minutes; a TypeScript dependency graph plus data from scheduled full-suite runs selects impacted tests. Scope: first-hand smaller-team case; not measured by Boat. How to use it: Affected-test selection needs a scheduled full run as the safety net. It maps to the `skip-unrelated-jobs` practice.
- Reported by Shopify engineering (https://shopify.engineering/faster-shopify-ci): p95 of the core monolith CI went from 45 to 18 minutes across more than 170,000 tests, after instrumenting the pipeline and changes such as caching schema setup. Scope: first-hand historical Buildkite case; not measured by Boat. How to use it: Instrument first. The step that dominates is often setup that every shard repeats.
- Reported by Anthropic engineering (https://claude.com/resources/articles/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic): CI job volume grew 25 times in six months; a test selector that uses prior results and package relevance was scaled to cope. Scope: first-hand engineering article about internal CI volume; not measured by Boat. How to use it: When volume is the problem, running fewer tests per change matters more than running the same tests on more machines.
- Reported by Vercel (https://vercel.com/blog/introducing-vercel-ci): 2 to 6 times faster push-to-green in internal repositories with graph-based affected tasks and caching. Scope: vendor launch post, internal benchmark; not measured by Boat. How to use it: A vendor number for its own product. Treat it as a direction to test, not an expected result.
- Reported by Daniel Lockyer (https://x.com/DanielLockyer/status/2117224680943690023): turn a GitHub Actions run into a Perfetto trace so an agent can see job timing. Scope: practitioner suggestion on X, no number. How to use it: Step 1 of section 5 is the same idea: look at the timeline before choosing what to split.
- Reported by Daniel Lockyer (https://x.com/DanielLockyer/status/2117279086875701528): consolidate short jobs when per-job minute rounding dominates the bill. Scope: practitioner suggestion on X; GitHub documents the rounding at https://docs.github.com/en/billing/reference/actions-runner-pricing. How to use it: More shards means more jobs, each billed in whole minutes. Do not shard a job that already finishes in well under a minute.
- Reported by Kun Chen (https://x.com/kunchenguid/status/2115565245293039814): balance load when build, lint and all tests run on every agent change. Scope: practitioner advice on X, no number. How to use it: Supports balancing by load. No measurement to quote.
- Reported by Arthur Katcher (https://x.com/arthurkatcher/status/2115832759159517368): discussion of removing unit tests from one team's CI. Scope: discussion on X with no measured evidence. How to use it: Not adopted here. The rule in this document stays: never remove or weaken tests to get a faster run.
- Reported by CircleCI documentation (https://circleci.com/docs/guides/optimize/parallelism-faster-jobs/): test splitting uses historical durations from saved test results. Scope: vendor documentation of a mechanism, no speedup number. How to use it: Another implementation of step 3 in section 5, if the pipeline runs there.

## 8. The loop

1. Fetch the report (section 1). Write down the baseline: `summary.wallClock.p50Seconds` and the critical-path jobs.
2. Take the suggestions with `applies: true`. Order them by `wallClockSavedSeconds`, then by `runnerHoursSavedPerDay`. Pick the first one that a workflow edit can deliver.
3. Open the file at `summary.workflowPath` and make that one change (section 3 says what to change; section 4 links real pull requests that did it; for `more-shards` and `balanced-shards` follow section 5 step by step). One practice per pull request, so its effect can be measured.
4. Open a pull request. Ask the person you work for before pushing if you have not been told you may.
5. Measure. Compare run time on the pull request with the baseline, using at least 3 successful runs on each side: `gh run list --workflow <file> --branch <branch> --status success --json createdAt,updatedAt`. Use GitHub run times for this, because the Boat report is reused for 24 hours.
6. Keep the change if the run is faster and still green. Otherwise revert it and say why.
7. Repeat from step 2 with the next suggestion. Stop when no remaining suggestion applies or the last two changes saved nothing.
8. Report: a table of each change with the estimated saving from Boat and the measured before and after, each number labelled estimated or measured.

Rules: never remove or weaken tests to get a faster run. Never change repository settings or secrets. Keep every number labelled as estimated, measured or reported by its author. Work only in the repository you were asked to work in: do not open pull requests, issues or comments on any other repository named in this document.
