CI map · public data

66m 28smedian of 184 sampled runs, up to 40 jobs at once · measured
58m 56swith Boat, up to 40 jobs at once · estimated

pandas-dev/pandas

Today the median run takes 66m 28s; resharded on Boat the plan estimates 58m 56s, 11% less wall clock. Estimated, not measured. Measured on boat: 77,286 tests (a subset of the suite) ran in 86.6 s on 1 machine and 17.2 s on 8 machines (runs below).

Today measured66m 28smedian of 184 runs · bars from run #44562983054 · up to 40 jobs at once
  1. Future configs legacy values44m 08s
  2. Future configs upcoming values42m 12s
  3. ubuntu-24.04 py31237m 42s
  4. windows-2025 py31236m 47s
  5. ubuntu-24.04 py31135m 03s
  6. windows-2025 py31134m 48s
  7. macos-15-intel py31365m 12s
  8. macos-15-intel py31258m 21s
  9. macos-15-intel py31146m 46s
  10. macos-15 py31136m 26s
  11. macos-15 py31235m 38s
  12. ubuntu-24.04-arm py31234m 17s
Show all 40 jobs
  1. Minimum Versions25m 49s
  2. Clang (ASAN/UBSAN)20m 22s
  3. Linux-Musl10m 12s
  4. Future configs legacy values44m 08s
  5. Future configs upcoming values42m 12s
  6. ubuntu-24.04 py31237m 42s
  7. windows-2025 py31236m 47s
  8. ubuntu-24.04 py31135m 03s
  9. windows-2025 py31134m 48s
  10. Locale: it_IT30m 58s
  11. ubuntu-24.04 py31430m 14s
  12. ubuntu-24.04 py31329m 53s
  13. Pyarrow Nightly26m 58s
  14. Clang (ASAN/LSAN)24m 47s
  15. windows-2025 py31420m 28s
  16. Numpy Nightly16m 44s
  17. python-dev (windows-2025)16m 24s
  18. Linux-32-bit15m 42s
  19. Without PyArrow15m 37s
  20. Pyodide build11m 14s
  21. python-dev (ubuntu-24.04)10m 47s
  22. Freethreading9m 43s
  23. Windows-MinGW7m 51s
  24. Downstream Compat6m 46s
  25. macos-15-intel py31365m 12s
  26. macos-15-intel py31258m 21s
  27. macos-15-intel py31146m 46s
  28. macos-15 py31136m 26s
  29. macos-15 py31235m 38s
  30. windows-2025 py31334m 16s
  31. macos-15 py31331m 21s
  32. Locale: zh_CN21m 11s
  33. python-dev (macos-15-intel)19m 37s
  34. macos-15 py31419m 19s
  35. python-dev (macos-15)13m 54s
  36. ubuntu-24.04-arm py31234m 17s
  37. ubuntu-24.04-arm py31330m 51s
  38. ubuntu-24.04-arm py31130m 46s
  39. macos-15-intel py31428m 01s
  40. ubuntu-24.04-arm py31420m 17s
Recommended with Boat estimated58m 56swall clock · up to 40 jobs at once
  1. Future configs legacy values34m 40s
  2. Future configs upcoming values33m 35s
  3. ubuntu-24.04 py31230m 04s
  4. windows-2025 py31233m 13s
  5. windows-2025 py31131m 25s
  6. macos-15-intel py31358m 52s
  7. macos-15-intel py31252m 41s
  8. macos-15-intel py31142m 14s
  9. macos-15 py31132m 54s
  10. macos-15 py31232m 11s
  11. windows-2025 py31330m 57s
  12. macos-15 py31328m 19s
Show all 40 jobs
  1. Minimum Versions18m 41s
  2. Clang (ASAN/UBSAN)14m 50s
  3. Linux-Musl5m 25s
  4. Future configs legacy values34m 40s
  5. Future configs upcoming values33m 35s
  6. ubuntu-24.04 py31230m 04s
  7. windows-2025 py31233m 13s
  8. ubuntu-24.04 py31127m 44s
  9. windows-2025 py31131m 25s
  10. Locale: it_IT22m 53s
  11. ubuntu-24.04 py31422m 21s
  12. ubuntu-24.04 py31323m 28s
  13. Pyarrow Nightly19m 11s
  14. Clang (ASAN/LSAN)16m 42s
  15. windows-2025 py31418m 29s
  16. Numpy Nightly11m 42s
  17. python-dev (windows-2025)14m 49s
  18. Linux-32-bit6m 55s
  19. Without PyArrow10m 57s
  20. Pyodide build7m 46s
  21. python-dev (ubuntu-24.04)6m 27s
  22. Freethreading5m 41s
  23. Windows-MinGW7m 05s
  24. Downstream Compat1m 26s
  25. macos-15-intel py31358m 52s
  26. macos-15-intel py31252m 41s
  27. macos-15-intel py31142m 14s
  28. macos-15 py31132m 54s
  29. macos-15 py31232m 11s
  30. windows-2025 py31330m 57s
  31. macos-15 py31328m 19s
  32. Locale: zh_CN14m 11s
  33. python-dev (macos-15-intel)17m 43s
  34. macos-15 py31417m 27s
  35. python-dev (macos-15)12m 33s
  36. ubuntu-24.04-arm py31227m 17s
  37. ubuntu-24.04-arm py31324m 20s
  38. ubuntu-24.04-arm py31124m 14s
  39. macos-15-intel py31425m 18s
  40. ubuntu-24.04-arm py31414m 57s

Times: medians of 184 sampled runs; bars: one real pull request run (#44562983054) from public GitHub Actions data; recommended plan estimated. Analyzed 2026-11-20.

Measured runs

On boat, 432 pandas test files (77,286 tests, a subset) ran in 86.6 s on 1 machine and 17.2 s on 8 forked machines. 8 machines ran 259,601 tests in 32.7 s.

Every run, failures, GitHub Actions and limits

Real runs of pandas-dev/pandas, one run each. Compare within one group only. Bars share one scale.

On boat · each run alone · measured · 4 test directories: frame, series, indexes, groupby (432 files), a subset

  1. 1× default1m 27s77,286 tests, 1 failed
  2. 8× default17s77,286 tests, 1 failed, shards balanced by test file size in bytes

On boat · each run alone · measured · whole pandas/tests tree except io/pytables (966 files), a subset

  1. 8× default1m 24s259,603 tests, 26 failed, shards balanced by measured seconds per test file from an earlier run
  2. 8× default33s259,601 tests, 26 failed, shards balanced by measured seconds per test file from an earlier run

On boat · each run alone · measured · whole pandas/tests tree except io/pytables (966 files) · killed before finishing

  1. 1× default2m 20skilled, no result
  2. 1× default3m 24skilled, no result

GitHub Actions · p50 from history · side by side, not like for like

  1. ubuntu-24.04 py312 (Unit Tests workflow)31m 33s

Not like for like: the GitHub job installs every optional dependency, runs the whole test tree and uploads coverage; the boat runs install the required dependencies plus pyarrow, numexpr and bottleneck, so the two are listed side by side, not as a like-for-like speedup.

Failed tests: 1 in every run of 4 test directories: frame, series, indexes, groupby (432 files); 26 in every run of whole pandas/tests tree except io/pytables (966 files).

  • every number is one run, not a median; boat 1 machine and boat N machines use the same warm snapshot, the same machine size (default: 4 vCPU, 8 GB) and the same pytest command
  • the 1-machine against 8-machine comparison covers a subset: 432 test files in 4 directories (77,286 tests); it says nothing about the whole CI run
  • the whole tree on one default machine was killed twice before finishing (exit code 137, after 138 s and 204 s), so the whole-tree runs have no 1-machine baseline and no speedup figure
  • no run is free of failures: 1 test fails in the 4-directory scope on 1 machine and on 8 machines alike (test_tz_localize_overflow_tzlocal), and 26 tests fail in the whole-tree scope in both 8-machine runs with different shard layouts (time zone and pyarrow conversion tests); the failures follow the environment, not the sharding, and were not investigated further
  • tests that need optional dependencies not installed here are skipped (13,391 skipped of 259,601 in the whole-tree scope); pandas/tests/io/pytables is left out because its conftest needs PyTables
  • markers follow the project's CI default: not slow, not network, not db, not single_cpu
  • wallClockSeconds is measured from dispatch to the last machine finishing and excludes forkSeconds, which is listed next to it; machineSeconds sums the test seconds of every machine and costUsd prices it at the boat default rate ($0.036 per machine hour)
  • the first 8-machine run of each scope was balanced by file size or by partial timings; the 32.7 s whole-tree run was balanced by seconds per test file measured in the 84.2 s run before it
More detail

What to change

Recommended: 30 machines, $0.378 per run estimated The recommended plan keeps the 9 jobs of the workflow, starts each from a warm snapshot and splits the test work of the slowest job over more machines: 30 machines against 30 today.

  • Fork from a warm snapshot instead of repeating setupestimated

    Each job on the critical path spends its first steps on checkout, tool setup and installs. A boat.dev sandbox forks from a snapshot that already holds that state, so the job starts at the first test step.

    Saves 6m 18s per run
  • Balance shards by measured test timeestimated

    A sharded job finishes when its slowest shard does. Splitting tests by recorded duration brings the slowest shard down to the mean shard time.

    Saves 7m 57s per run
  • Run the critical-path tests on 12 machinesestimated

    Sandboxes are billed per second, so the same test work spread over more machines costs about the same and finishes sooner. Only per-shard setup stays constant.

    Saves 38m 15s per run
  • Cache dependencies between runsestimated

    The workflow reinstalls dependencies on every job. Restoring them from a cache removes part of the setup time of every job.

    82.8 runner hours a day
  • Retry failed tests in parallel inside the same runestimated

    A rerun repeats the whole workflow after someone notices the failure. Retrying only the failed tests on a fresh fork while the run is still going removes that second cycle.

    Saves 60 s per run
  • Test merges speculatively before they landestimated

    A red default branch costs at least one more full CI cycle to fix or revert. Testing each merge result on a fork before it lands keeps those failures off the default branch.

    Saves 1m 42s per run

How this CI evolved

Daily wall time of successful runs, p50 and p95, 2026-11-16 to 2026-11-20.

2026-11-162026-11-20

p50 p95

What's next: the same climb, automatic

pandas-dev/pandas can climb the same hill. Boat's agent will run that loop for you: try a plan, measure it, keep what is faster, try again. This is not live yet; when measured rounds exist, they will appear on this page.

See your own CI