refactor!: publish numbers, not a page

[zkochan]

This repository measured the benchmarks and also decided how they were presented — it wrote benchmarks.md and the charts it referred to, and pnpm.io copied both across. That put every wording change (a column renamed, a paragraph rewritten, a manager dropped from the comparison) behind a benchmark run in a repository whose job is measurement.

It now publishes one file, benchmarks.json: the numbers, the version of every tool they were measured with, and the conditions they were measured under — the pnpr version, the emulated round trip and bandwidth, the Node.js versions the version-manager section pins. Everything a reader is shown is decided in pnpm/pnpm.io, which renders the page and its charts from that file.

The min() over samples stays here. The statistic belongs with the measurement: only the code that took the samples knows what they are samples of.

What is measured doesn't change

Every manager in commandsMap.js is still measured and still reported, Yarn and Bun included. Whether any of them reaches a reader is now a separate question, answered downstream: the site draws npm and the pnpm columns, and the rest are measured for comparisons of our own.

That is the point of the split, and pnpm/pnpm.io#904 demonstrates it — the page renders byte-identically whether or not the manifest carries Yarn and Bun.

Gone

benchmarks.md, img/benchmarks/, generateSvg.js, generateStackedSvg.js, nodeVersionsSection.js, regenerate-svgs.mjs, and five dependencies only the rendering used. commandsMap.js and nodeManagersMap.js lost their legend / mdLegend / color / mascot fields — presentation, not measurement.

Before merging

• The committed benchmarks.json is built from the results already recorded here, so the site has something to render before the next run. Its pnpr and node values were read off the last published chart, since versions.json isn't committed; the next pnpm run report rewrites the file from the run's own manifest.
• pnpm-lock.yaml still carries the five removed dependencies — needs a pnpm install.
• Merge this before refactor: render the benchmark page here, from published numbers pnpm.io#904, which reads benchmarks.json from main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SKeHCgKWBLVERXWMdVHQFJ

[coderabbitai]

Warning

Review limit reached

Next included review available in 37 minutes.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 618d22c2-8f73-49c4-aa86-55fbe589c086

📥 Commits

Reviewing files that changed from the base of the PR and between 535d67c and d168edb.

⛔ Files ignored due to path filters (4)

• img/benchmarks/alotta-files-pnpm.svg is excluded by !**/*.svg
• img/benchmarks/alotta-files.svg is excluded by !**/*.svg
• img/benchmarks/node-versions.svg is excluded by !**/*.svg
• pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (12)

• .github/workflows/benchmark.yml
• README.md
• benchmarks.json
• benchmarks.md
• commandsMap.js
• generateStackedSvg.js
• generateSvg.js
• index.js
• nodeManagersMap.js
• nodeVersionsSection.js
• package.json
• regenerate-svgs.mjs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-code-review]

PR Summary by Qodo

Publish benchmark measurements as JSON instead of rendered pages

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Publish results, tool versions, and measurement conditions through benchmarks.json.
• Move benchmark page and chart presentation ownership to pnpm.io.
• Stop measuring Yarn and Bun, removing their specialized setup and dependencies.

Diagram

graph TD
  CI["CI Sample Jobs"] --> Harness["Benchmark Harness"] --> Tools["npm and pnpm"]
  Harness --> Samples[("Recorded Samples")] --> Report["Report Mode"] --> Manifest[("benchmarks.json")] --> Site["pnpm.io Renderer"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Dual-publish JSON and legacy pages

• ➕ Provides a migration window for existing consumers
• ➕ Allows direct comparison between old and new renderers
• ➖ Retains duplicated presentation ownership
• ➖ Keeps chart dependencies and manager metadata in the measurement repository
• ➖ Creates two outputs that can diverge

Recommendation: Use the PR's direct JSON-only cutover, coordinated with the pnpm.io consumer. Dual publishing would reduce migration risk but undermine the central separation of measurement from presentation and prolong maintenance of obsolete rendering code.

Files changed (11) +278 / -481

Enhancement (2) +217 / -321

benchmarks.jsonAdd the presentation-neutral benchmark manifest +102/-0

Add the presentation-neutral benchmark manifest

• Commits initial benchmark minima with measurement time, Node.js and pnpr versions, emulated network conditions, fixture results, and Node.js version-manager results. This becomes the sole published artifact consumed by pnpm.io.

benchmarks.json

index.jsReplace page rendering with JSON manifest generation +115/-321

Replace page rendering with JSON manifest generation

• Refactors measurement and report-only flows to collect minimum scenario timings and write 'benchmarks.json' with tool versions and benchmark conditions. Removes markdown, SVG, formatting, sorting, hashing, and Yarn/Bun orchestration while preserving sample reduction in the measurement repository.

index.js

Refactor (6) +42 / -139

benchmarkFixture.jsRemove Yarn and Bun fixture lifecycle handling +25/-70

Remove Yarn and Bun fixture lifecycle handling

• Drops Yarn and Bun lockfile mappings, Yarn configuration generation, PnP cleanup, and rewarm-file handling. Simplifies lockfile and install-output cleanup for the remaining npm and pnpm scenarios.

benchmarkFixture.js

benchmarkRegistry.jsSimplify registry overrides for retained managers +2/-5

Simplify registry overrides for retained managers

• Removes Yarn's registry-flag exemption now that every measured package manager accepts the benchmark registry argument.

benchmarkRegistry.js

commandsMap.jsRestrict install benchmarks to npm and pnpm +7/-41

Restrict install benchmarks to npm and pnpm

• Removes Yarn and Bun command definitions and strips legends, colors, display versions, and mascot metadata from retained managers. The map now contains only measurement commands and scenario behavior.

commandsMap.js

nodeManagersMap.jsRemove presentation metadata from Node manager definitions +1/-9

Remove presentation metadata from Node manager definitions

• Retains only executable and scenario identities for pnpm, fnm, and nvm, leaving labels and colors to the downstream renderer.

nodeManagersMap.js

pnprServer.jsUpdate pnpr installation guidance for retained tools +1/-1

Update pnpr installation guidance for retained tools

• Removes the obsolete Bun comparison from the native-binary installation comment.

pnprServer.js

setupPackageManagers.jsStop provisioning Yarn and Bun +6/-13

Stop provisioning Yarn and Bun

• Limits package-manager provisioning to npm and pnpm 11/12 and updates safety documentation accordingly. Removes Yarn 6 and Bun global installation steps.

setupPackageManagers.js

Documentation (1) +6 / -4

README.mdDocument the measurement-only repository contract +6/-4

Document the measurement-only repository contract

• Points readers to 'benchmarks.json' and the pnpm.io benchmark page. Explains manifest contents, minimum-sample selection, report-only generation, and the reduced npm/pnpm manager scope.

README.md

Other (2) +13 / -17

benchmark.ymlAlign CI reporting with JSON publication +13/-10

Align CI reporting with JSON publication

• Updates workflow documentation to describe 'benchmarks.json' generation and the current pnpm setup constraint. Removes the GitHub token previously required for Yarn release discovery.

.github/workflows/benchmark.yml

package.jsonRemove rendering and retired-manager dependencies +0/-7

Remove rendering and retired-manager dependencies

• Deletes the SVG regeneration script and six packages used by page rendering or Yarn-specific fixture handling. Keeps only dependencies required for measurement, result loading, and fixture management.

package.json