Case study · public data
T3 Code tunes its CI by hand.
Today the median run takes 3m 28s; resharded on Boat the plan estimates 3m 05s, 11% less wall clock. Estimated, not measured. Measured so far on boat: 8 machines, 402 s for the full CI (details below).
- Test Server 61m 54s
- Test Server 51m 36s
- Test Server 11m 32s
- Lint1m 00s
- Changes7 s
- Test Server 23m 13s
- Test Server 81m 28s
- Test Server 42m 31s
- Test Server 32m 23s
- Test Server 72m 16s
- Build and Typecheck48 s
- Test37 s
Show all 13 jobs
- Test Server 61m 54s
- Test Server 51m 36s
- Test Server 11m 32s
- Lint1m 00s
- Changes7 s
- Test Server 23m 13s
- Test Server 81m 28s
- Test Server 42m 31s
- Test Server 32m 23s
- Test Server 72m 16s
- Build and Typecheck48 s
- Test37 s
- Check3 s
- Test Server 61m 41s
- Test Server 51m 22s
- Test Server 11m 18s
- Lint45 s
- Changes8 s
- Test Server 22m 54s
- Test Server 81m 15s
- Test Server 42m 14s
- Test Server 32m 07s
- Test Server 71m 58s
- Build and Typecheck31 s
- Test19 s
Show all 13 jobs
- Test Server 61m 41s
- Test Server 51m 22s
- Test Server 11m 18s
- Lint45 s
- Changes8 s
- Test Server 22m 54s
- Test Server 81m 15s
- Test Server 42m 14s
- Test Server 32m 07s
- Test Server 71m 58s
- Build and Typecheck31 s
- Test19 s
- Check3 s
Times: medians of 157 sampled runs; bars: one real pull request run (#44798501187) from public GitHub Actions data; recommended plan estimated. Analyzed 2026-11-20.
More detail
What a pull request waits for
Weekly wall time of successful pull request runs of ci.yml, measured. Each dot is a performance change; hover or tap one.
- PR CI p50
- PR CI p95
The last 30 days (Oct 21 – Nov 20, 2026), measured from public GitHub data
What they changed
17 performance changes to ci.yml, read from the repository's git history, in 6 techniques.
- Faster machines2 changesMove to Blacksmith runners #111
- Split jobs1 changeSplit the single job into parallel jobs #2940
- Shard tests4 changesParallelize tests, split out Rust checks #7286
- Cache4 changesCache Chromium for server test shards #19632
- Skip work3 changesCancel stale runs, drop redundant setup #4802
- Shorten critical path3 changesRun typecheck inside the Build job #19633
Measured runs
Real runs of pingdotgg/t3code. Compare within one group only. Bars share one scale.
9 measured runs of t3code. Fastest on boat: 8 machines, 6m 42s (25/25 CI tasks).
Every run and its limits
Local Docker, not boat · four tiers at the same time · measured
- 1× default22m 1s8/10 CI tasks
- 2× default11m 58s9/12 CI tasks
- 4× default9m 3s12/16 CI tasks
- 8× default6m 50s21/25 CI tasks
Local Docker, not boat · each run alone · measured
- 1× default19m 56s9/10 CI tasks
- 8× default5m 53s20/25 CI tasks
On boat · each run alone · measured
- 4× default8m 4s16/16 CI tasks
- 8× default6m 42s25/25 CI tasks
- 1× large · every step in a row18m 37s
GitHub Actions · p50 from history
- Pull request CI, p503m 22s
- compare like with like: local-docker-alone 1x against local-docker-alone 8x, boat against boat; local-docker-concurrent tiers (1, 2, 4, 8 machines) ran at the same time on two 32-thread hosts, 60 of 64 threads requested, so each tier shares memory bandwidth and disk with the others and is slower than it would be alone
- two tests fail on both backends outside GitHub Actions: AntigravityInstallation (apps/server) and CloudVoiceConfiguration (apps/web, 1 of 7067); passed=false reflects that; passedExceptEnvironment and environmentSkipped list them by name. Which shard held them is inferred from the failed task and the single-machine logs
- on local Docker the rust task failed for a missing rustfmt component and the packages test for a default ACL that changes file modes; local-docker-1x-default-alone runs with both fixed, the earlier local runs carry them in environmentSkipped
- local-docker-1x-default-alone ran with the rustfmt and tmpfs fixes, yet its packages test task still failed; the failing test was not captured
- boat-8x-default and boat-4x-default: the dispatcher marked the sync step (checkout + offline install, 23-40 s) failed on every machine, so passed=false; every CI task then ran on that tree and exited 0 (ciTasksPassed 25/25 and 16/16). The same sync passes when rerun by hand on the source machine; the cause is not confirmed
- boat fork = boat fork of a running warm machine (zero data retention blocks saved snapshots) until the first exec succeeds; the fork number in boat-8x is the median over 8 parallel forks, including the CLI round trips
- local fork = docker run of the warm image until the first exec succeeds, median of 5 on ubuntu05
- the local single-machine table covers setup and build only; the boat single-machine table covers every step on a large (8 vCPU) machine
- local-docker runs use containers limited to the boat size (default: 4 CPUs, 8 GB) on 32-core hosts; their costUsd is what the same machine seconds would cost at the boat rate, not money spent
- Mobile Native Static Analysis (macOS) is not run: neither backend has macOS
- the task cache, the pnpm store cache and the Vitest module cache that GitHub Actions restores are not used; the warm snapshot replaces them
What to change
Recommended: 28 machines, $0.015 per run estimated The recommended plan keeps the 10 jobs of the workflow, starts each from a warm snapshot and splits the test work of the slowest job over more machines: 28 machines against 18 today.
- Fork from a warm snapshot instead of repeating setupestimated
Each job on the critical path spends its first steps on checkout, tool setup and installs. A boat.dev sandbox forks from a snapshot that already holds that state, so the job starts at the first test step.
Saves 19 s per run - Balance shards by measured test timeestimated
A sharded job finishes when its slowest shard does. Splitting tests by recorded duration brings the slowest shard down to the mean shard time.
Saves 21 s per run - Run the critical-path tests on 12 machinesestimated
Sandboxes are billed per second, so the same test work spread over more machines costs about the same and finishes sooner. Only per-shard setup stays constant.
Saves 1m 38s per run
What's next: the same climb, automatic
T3 Code climbed one pull request at a time. Boat's agent will run that loop for you: try a plan, measure it, keep what is faster, try again. This is not live yet; when measured rounds exist, they will appear on this page.