Case study · public data

3m 28smedian of 157 sampled runs, up to 11 jobs at once · measured
3m 05swith Boat, up to 11 jobs at once · estimated

T3 Code tunes its CI by hand.

Today the median run takes 3m 28s; resharded on Boat the plan estimates 3m 05s, 11% less wall clock. Estimated, not measured. Measured so far on boat: 8 machines, 402 s for the full CI (details below).

Today measured3m 28smedian of 157 runs · bars from run #44798501187 · up to 11 jobs at once
  1. Test Server 61m 54s
  2. Test Server 51m 36s
  3. Test Server 11m 32s
  4. Lint1m 00s
  5. Changes7 s
  6. Test Server 23m 13s
  7. Test Server 81m 28s
  8. Test Server 42m 31s
  9. Test Server 32m 23s
  10. Test Server 72m 16s
  11. Build and Typecheck48 s
  12. Test37 s
Show all 13 jobs
  1. Test Server 61m 54s
  2. Test Server 51m 36s
  3. Test Server 11m 32s
  4. Lint1m 00s
  5. Changes7 s
  6. Test Server 23m 13s
  7. Test Server 81m 28s
  8. Test Server 42m 31s
  9. Test Server 32m 23s
  10. Test Server 72m 16s
  11. Build and Typecheck48 s
  12. Test37 s
  13. Check3 s
Recommended with Boat estimated3m 05swall clock · up to 11 jobs at once
  1. Test Server 61m 41s
  2. Test Server 51m 22s
  3. Test Server 11m 18s
  4. Lint45 s
  5. Changes8 s
  6. Test Server 22m 54s
  7. Test Server 81m 15s
  8. Test Server 42m 14s
  9. Test Server 32m 07s
  10. Test Server 71m 58s
  11. Build and Typecheck31 s
  12. Test19 s
Show all 13 jobs
  1. Test Server 61m 41s
  2. Test Server 51m 22s
  3. Test Server 11m 18s
  4. Lint45 s
  5. Changes8 s
  6. Test Server 22m 54s
  7. Test Server 81m 15s
  8. Test Server 42m 14s
  9. Test Server 32m 07s
  10. Test Server 71m 58s
  11. Build and Typecheck31 s
  12. Test19 s
  13. Check3 s

Times: medians of 157 sampled runs; bars: one real pull request run (#44798501187) from public GitHub Actions data; recommended plan estimated. Analyzed 2026-11-20.

More detail

What a pull request waits for

Weekly wall time of successful pull request runs of ci.yml, measured. Each dot is a performance change; hover or tap one.

  • PR CI p50
  • PR CI p95
0s5m10mMarAprMayJunJulAugSepOctNov
ci.yml

The last 30 days (Oct 21 – Nov 20, 2026), measured from public GitHub data

PR CI p50
3.4 min
p90 4.9 min. Sharding tests by measured duration could reach about 1.3–1.5 min estimated, not measured
Main commits that ended with red CI
5.7%
306 of 5,344. Main has no merge queue and does not require branches to be up to date before merging.
PR CI runs on a second attempt or later
19%
Mostly fork pull requests waiting for a maintainer to approve the run, not flaky reruns.

What they changed

17 performance changes to ci.yml, read from the repository's git history, in 6 techniques.

  • Faster machines2 changesMove to Blacksmith runners #111
  • Split jobs1 changeSplit the single job into parallel jobs #2940
  • Shard tests4 changesParallelize tests, split out Rust checks #7286
  • Cache4 changesCache Chromium for server test shards #19632
  • Skip work3 changesCancel stale runs, drop redundant setup #4802
  • Shorten critical path3 changesRun typecheck inside the Build job #19633

Measured runs

Real runs of pingdotgg/t3code. Compare within one group only. Bars share one scale.

9 measured runs of t3code. Fastest on boat: 8 machines, 6m 42s (25/25 CI tasks).

Every run and its limits

Local Docker, not boat · four tiers at the same time · measured

  1. 1× default22m 1s8/10 CI tasks
  2. 2× default11m 58s9/12 CI tasks
  3. 4× default9m 3s12/16 CI tasks
  4. 8× default6m 50s21/25 CI tasks

Local Docker, not boat · each run alone · measured

  1. 1× default19m 56s9/10 CI tasks
  2. 8× default5m 53s20/25 CI tasks

On boat · each run alone · measured

  1. 4× default8m 4s16/16 CI tasks
  2. 8× default6m 42s25/25 CI tasks
  3. 1× large · every step in a row18m 37s

GitHub Actions · p50 from history

  1. Pull request CI, p503m 22s
  • compare like with like: local-docker-alone 1x against local-docker-alone 8x, boat against boat; local-docker-concurrent tiers (1, 2, 4, 8 machines) ran at the same time on two 32-thread hosts, 60 of 64 threads requested, so each tier shares memory bandwidth and disk with the others and is slower than it would be alone
  • two tests fail on both backends outside GitHub Actions: AntigravityInstallation (apps/server) and CloudVoiceConfiguration (apps/web, 1 of 7067); passed=false reflects that; passedExceptEnvironment and environmentSkipped list them by name. Which shard held them is inferred from the failed task and the single-machine logs
  • on local Docker the rust task failed for a missing rustfmt component and the packages test for a default ACL that changes file modes; local-docker-1x-default-alone runs with both fixed, the earlier local runs carry them in environmentSkipped
  • local-docker-1x-default-alone ran with the rustfmt and tmpfs fixes, yet its packages test task still failed; the failing test was not captured
  • boat-8x-default and boat-4x-default: the dispatcher marked the sync step (checkout + offline install, 23-40 s) failed on every machine, so passed=false; every CI task then ran on that tree and exited 0 (ciTasksPassed 25/25 and 16/16). The same sync passes when rerun by hand on the source machine; the cause is not confirmed
  • boat fork = boat fork of a running warm machine (zero data retention blocks saved snapshots) until the first exec succeeds; the fork number in boat-8x is the median over 8 parallel forks, including the CLI round trips
  • local fork = docker run of the warm image until the first exec succeeds, median of 5 on ubuntu05
  • the local single-machine table covers setup and build only; the boat single-machine table covers every step on a large (8 vCPU) machine
  • local-docker runs use containers limited to the boat size (default: 4 CPUs, 8 GB) on 32-core hosts; their costUsd is what the same machine seconds would cost at the boat rate, not money spent
  • Mobile Native Static Analysis (macOS) is not run: neither backend has macOS
  • the task cache, the pnpm store cache and the Vitest module cache that GitHub Actions restores are not used; the warm snapshot replaces them

What to change

Recommended: 28 machines, $0.015 per run estimated The recommended plan keeps the 10 jobs of the workflow, starts each from a warm snapshot and splits the test work of the slowest job over more machines: 28 machines against 18 today.

  • Fork from a warm snapshot instead of repeating setupestimated

    Each job on the critical path spends its first steps on checkout, tool setup and installs. A boat.dev sandbox forks from a snapshot that already holds that state, so the job starts at the first test step.

    Saves 19 s per run
  • Balance shards by measured test timeestimated

    A sharded job finishes when its slowest shard does. Splitting tests by recorded duration brings the slowest shard down to the mean shard time.

    Saves 21 s per run
  • Run the critical-path tests on 12 machinesestimated

    Sandboxes are billed per second, so the same test work spread over more machines costs about the same and finishes sooner. Only per-shard setup stays constant.

    Saves 1m 38s per run

What's next: the same climb, automatic

T3 Code climbed one pull request at a time. Boat's agent will run that loop for you: try a plan, measure it, keep what is faster, try again. This is not live yet; when measured rounds exist, they will appear on this page.

See your own CI