Skip to content

RFC: Cargo ecosystem — pnpm for Rust crates, pnpr as a crate registry - #34

Draft
zkochan wants to merge 13 commits into
mainfrom
rust-pm
Draft

RFC: Cargo ecosystem — pnpm for Rust crates, pnpr as a crate registry#34
zkochan wants to merge 13 commits into
mainfrom
rust-pm

Conversation

@zkochan

@zkochan zkochan commented Sep 4, 2026

Copy link
Copy Markdown
Member

Summary

Adds text/0000-cargo-ecosystem.md, an RFC that makes pnpm a package manager for Rust crates and pnpr a crate registry.

Why: since the CLI rewrite in Rust, pnpm no longer dogfoods itself. The Rust workspace's 714 crates.io dependencies (811 lockfile entries) are fetched by cargo, cached by Swatinem/rust-cache, audited by cargo-deny and bumped by Dependabot. None of it touches pnpm or pnpr.

What the RFC proposes:

  • pnpm resolves, fetches, verifies and materializes crates; cargo only compiles. Crates are hardlinked from the existing store into .pnpm/crates/<source>/ as cargo directory sources, wired up through a pnpm-managed region of a committed .cargo/config.toml.
  • Cargo.lock stays the one lockfile for crates, written in cargo's canonical form and accepted by cargo --locked.
  • Independent ecosystem installers run concurrently and share one configured HTTP client, preserving the aggregate network limits; the coordinator is designed for later languages such as Python/PyPI.
  • A new PubGrub-based resolver crate with cargo's semantics (one version per semver-compatible range, backtracking, feature unification, MSRV, yank rules), tested differentially against cargo's resolver, starting with reproducing this repository's own lockfile byte for byte.
  • Opt-in via cargo.enabled in pnpm-workspace.yaml; cargo's [registries] and credentials.toml are honoured rather than duplicated.
  • A crate: specifier protocol across add/remove/update/outdated/list/why/audit/licenses/publish/pack, with [workspace.dependencies] as the crate catalog.
  • pnpr gains protocol: cargo on registries under the existing mounts model: sparse index, download, publish/yank/owners/search web API, bare-token auth, and an upstream mode mirroring crates.io.
  • A four-milestone dogfooding plan for pnpm/pnpm, where milestone 1 (pnpr proxying crates.io for plain cargo) needs no pnpm changes.

Decided: new ecosystem features ship in the Rust pnpm v12 CLI, following the repository's v12-only feature-development policy.

The architecture section records the real Python integration in pnpm/pnpm#14586: opt-in install/add/run/exec support, PEP 440/508 resolution, native wheels, shared HTTP/auth/CAS, and staged environment publication. Python retains standard pylock.toml; uv independently installed a generated real-PyPI lockfile. The integration now shares a dedicated lifecycle crate across install and mixed add: native metadata ownership, concurrent preparation, settlement, publication and reverse rollback. Cargo and Python return prepared projections; npm retains explicit in-place semantics. Target identity remains native, and the boundary is internal rather than a public plugin API. Source builds, universal locking and crash recovery remain outside this slice.

A duplication audit also consolidated tarball and ZIP cache validation, offline gating, HTTP request setup, extraction retries, progress and store publication. npm packages, Cargo crates, Python wheels and runtime binaries use these same implementations. Container decoding and native dependency semantics remain separate, with cross-format contract tests guarding the common behavior.

Open questions (naming, --optional, whether to split the pnpr half into pnpr/text/) are listed at the end of the document.

🤖 Generated with Claude Code, then continued with Codex.

https://claude.ai/code/session_0167jVFmSFDDzaTKAvCEbC1Y


Updated by an agent (Codex, GPT-5).

The Rust CLI's own dependencies are fetched by cargo from crates.io and
never pass through pnpm or pnpr. This RFC adds a second ecosystem to
pnpm (resolution, fetching, store, Cargo.lock, directory-source
materialization) and a `cargo` registry protocol to pnpr (sparse index,
download, web API, crates.io upstream), with a dogfooding plan for the
pnpm repository. Ships in pacquet only, as a decided exception to the
parity rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0167jVFmSFDDzaTKAvCEbC1Y
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

zkochan and others added 2 commits September 4, 2026 15:37
…kspace trees

Crate slots are unpacked once per machine under `<store>/crates/`;
each workspace holds one symlink per locked crate in its directory
sources. Verified against cargo 1.97 that directory sources follow
symlinked entries, ignore dot-prefixed ones, and verify checksums at
compile time. Records the per-workspace hardlinked tree as a rejected
alternative and adds a self-contained layout as an open question.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0167jVFmSFDDzaTKAvCEbC1Y
@zkochan

zkochan commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Recorded the architecture exposed by the Cargo proof of concept in 688d198: an ecosystem-neutral install kernel, ecosystem-owned resolver and lockfile semantics, staged cross-ecosystem changes, normalized auth routing, and an explicit zero-regression requirement for npm-only installs enforced by the existing integrated benchmarks.


Written by an agent (Codex, gpt-5.6-sol).

@zkochan

zkochan commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

The verified-artifact ingestion slice has landed in pnpm/pnpm#14574. Commit ff49aaa updates this RFC with the implementation finding that integrity identifies immutable bytes, while projection inputs identify the extracted file map and therefore belong in every memory/disk cache key. It also records the legacy-row migration rules that fell out of review.


Written by an agent (Codex, gpt-5.6-sol).

@zkochan

zkochan commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Implementation update: pnpm/pnpm#14578 merged and the RFC now records the request-auth invariants learned from it.

The important boundary is that the shared network layer routes already-formatted headers by URL. npm and Cargo configuration readers retain their own formats, schemes, and precedence rules. Cargo credential discovery happens once and only on the Cargo path; redirect destinations are authorized independently; parse failures do not echo credential contents. The current crates.io token is deliberately scoped to the index host, while named/private registry mapping remains a later Cargo source-identity slice.


Written by an agent (Codex, gpt-5.6-sol).

Document the real Python CLI integration in pnpm/pnpm#14586 and its validation of shared network, auth, and artifact storage. Record staged environment publication, native pylock interoperability, and remaining compatibility limits.
zkochan added a commit to pnpm/pnpm that referenced this pull request Sep 5, 2026
Route Python projects through production install and add coordination.
Keep PEP 440/508 resolution and pylock.toml native to Python while sharing
the HTTP/auth budget, verified ZIP ingestion, CAS, store index and reporter.

Prepare complete virtual environments before publication, and retain mixed
add metadata rollback. Include Python executables in run and exec paths.
Exercise real wheels, native extensions, offline/frozen replay, and failures.

Record the staged-publication and target-identity requirements in the RFC.
New Python features are opt-in and target pnpm v12 only.

Related to #14566 and pnpm/rfcs#34.
zkochan added a commit to pnpm/pnpm that referenced this pull request Sep 5, 2026
Extract workspace locking, metadata snapshots, concurrent preparation,
settlement and publication into a dedicated install-coordinator crate.
The lifecycle has no dependency on CLI configuration, ecosystem identity,
resolver semantics or native lockfile formats.

Let Cargo and Python declare metadata footprints and return prepared
projections. Publish only after every participant settles. Reverse all
attempted publications on failure, then restore metadata. Retain resources
when rollback fails so recovery does not reference deleted generations.

Move Cargo add preparation into its adapter and remove native file lists
and Python-specific publication state from command pipelines. Sort Cargo
results for deterministic publication despite concurrent preparation.
Drain Cargo work before returning errors, including blocking store tasks.

Exclude configured stores and caches from native workspace discovery before
descent. Repeated installs must not reinterpret cached native manifests as
workspace inputs. Keep unrelated projects with the same basenames visible.

Keep npm's in-place materialization and lifecycle-script semantics explicit.
Mixed add enrolls its metadata in rollback. The npm-only path creates no
plan, inventory or transaction. No universal resolver or lockfile is added.

Cover real cross-ecosystem preparation/publication failures and shared
rollback/resource-lifetime contracts. Match Python's native Windows CRLF
in the existing entry-point assertion found by remote CI.

Related to #14566 and pnpm/rfcs#34.
zkochan added a commit to pnpm/pnpm that referenced this pull request Sep 5, 2026
Route tarball and ZIP ingestion through one cache and publication lifecycle.
Share authenticated request setup, concurrency permits, HTTP status handling,
bounded error-body draining, progress tracking and extraction retries.
Keep tar streaming and ZIP decoding as container-specific strategies.

Preserve npm cache keys, projection namespaces, revision-addressed requests,
local tarball support and the existing in-memory hit fast path. The native
adapters continue to own dependency rules and installed filesystem layouts.
Input adapters borrow install-scoped handles without adding reference-count
updates when constructing the shared ingestion request.

Use the resolved retry policy for npm, Cargo and Python artifact requests.
Replace Python's independent metadata retry loop with the shared network
policy and apply it to Cargo sparse-index requests. Cover interrupted bodies,
HTTP 408, credential isolation on redirects and credential-free diagnostics.

Add cross-format contracts for cache validation, offline replay, projection
selection, retry classification and unpublished corrupt transfers. Document
the ownership audit without claiming that native algorithms are equivalent.

Related to #14566 and pnpm/rfcs#34.
zkochan added a commit to pnpm/pnpm that referenced this pull request Sep 5, 2026
…es (#14586)

Exercise the multi-ecosystem architecture through a usable Python
integration, not test-only metadata writers.

Keep Python requirement, marker, lockfile and environment semantics
separate from npm and Cargo. Reuse the install-wide HTTP/auth budget,
verified artifact ingestion, CAS and store index.

Extract a shared install lifecycle without ecosystem-specific branches.
Native tasks declare metadata footprints and return prepared projections.
Settle all work before publishing Cargo or Python state, and reverse
attempted publications before restoring metadata on failure.
Retain resources when rollback fails so recovery remains possible.

Enroll npm with its existing in-place materialization semantics, and
keep npm-only dispatch on its early path. Do not unify native resolvers,
lockfile formats, target identity or package layouts.

Consolidate archive ingestion below the ecosystem boundary. Share cache
validation, authenticated requests, extraction retries and publication
across tarballs and ZIPs, retaining tar streaming and ZIP decoding.
Test the shared contracts across both formats and preserve npm fast paths.

Use standard pylock.toml, independently validated with uv.
New features target pnpm v12 only. Shared archive URL-redaction fixes
also cover pnpm v11.

Related to #14566 and pnpm/rfcs#34.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant