Skip to content

Add retry support to sparse registries - #11069

Merged
bors merged 1 commit into
rust-lang:masterfrom
arlosi:retry
Oct 7, 2022
Merged

bors merged 1 commit into
rust-lang:masterfrom
arlosi:retry

Conversation

@arlosi

@arlosi arlosi commented Sep 9, 2022

Copy link
Copy Markdown
Contributor

Sparse (HTTP) registries currently do not respect Cargo's retry policy for http requests.

This change makes sparse registries use the same retry system as package downloads.

@rust-highfive

Copy link
Copy Markdown

r? @epage

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 9, 2022

@epage epage left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've taken a look and it seems good but I would want @Eh2406 to take a look as I assume they have more recently reviewed this area of code.

Comment thread src/cargo/sources/registry/http_remote.rs Outdated
@epage

epage commented Sep 9, 2022

Copy link
Copy Markdown
Contributor

r? @Eh2406

@rust-highfive rust-highfive assigned Eh2406 and unassigned epage Sep 9, 2022
@bors

bors commented Sep 9, 2022

Copy link
Copy Markdown
Contributor

☔ The latest upstream changes (presumably #11068) made this pull request unmergeable. Please resolve the merge conflicts.

@Eh2406

Eh2406 commented Sep 11, 2022

Copy link
Copy Markdown
Contributor

Had some time to look this over and it looks good to me. Ed has some good questions about nomenclature. Is there some way we can add tests? Perhaps a setup like #11062 where the server takes a few tries to give a 200.

@weihanglo weihanglo added the A-sparse-registry Area: http sparse registries label Sep 13, 2022
@ehuss ehuss added S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 25, 2022
@arlosi

arlosi commented Oct 7, 2022

Copy link
Copy Markdown
Contributor Author

I pulled in the test framework changes from #11062 and added a test.

Comment thread src/cargo/util/mod.rs
pub fn truncate_with_ellipsis(s: &str, max_width: usize) -> String {
// We should truncate at grapheme-boundary and compute character-widths,
// yet the dependencies on unicode-segmentation and unicode-width are
// not worth it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unicode-width is already a dependency, maybe we should open an issue asking for this function to be updated to use it.

@Eh2406

Eh2406 commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

@bors r+

@bors

bors commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

📌 Commit dcc512b has been approved by Eh2406

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. labels Oct 7, 2022
@bors

bors commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

⌛ Testing commit dcc512b with merge d1a2592...

@bors

bors commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

☀️ Test successful - checks-actions
Approved by: Eh2406
Pushing d1a2592 to master...

@bors
bors merged commit d1a2592 into rust-lang:master Oct 7, 2022
weihanglo added a commit to weihanglo/rust that referenced this pull request Oct 11, 2022
9 commits in 3cdf1ab25dc4fe56f890e8c7330d53a23ad905d3..b8f30cb23c4e5f20854a4f683325782b7cff9837
2022-10-07 17:34:03 +0000 to 2022-10-10 19:16:06 +0000

- Add more doc comments for three modules (rust-lang/cargo#11207)
- docs: fix (rust-lang/cargo#11208)
- Add completions for `cargo remove` (rust-lang/cargo#11204)
- Config file loaded via CLI takes priority over env vars (rust-lang/cargo#11077)
- Use `#[default]` when possible (rust-lang/cargo#11197)
- Implement RFC 3289: source replacement ambiguity (rust-lang/cargo#10907)
- Use correct version of cargo in test (rust-lang/cargo#11193)
- Check empty input for login (rust-lang/cargo#11145)
- Add retry support to sparse registries (rust-lang/cargo#11069)
bors added a commit to rust-lang-ci/rust that referenced this pull request Oct 12, 2022
Update cargo

9 commits in 3cdf1ab25dc4fe56f890e8c7330d53a23ad905d3..b8f30cb23c4e5f20854a4f683325782b7cff9837 2022-10-07 17:34:03 +0000 to 2022-10-10 19:16:06 +0000

- Add more doc comments for three modules (rust-lang/cargo#11207)
- docs: fix (rust-lang/cargo#11208)
- Add completions for `cargo remove` (rust-lang/cargo#11204)
- Config file loaded via CLI takes priority over env vars (rust-lang/cargo#11077)
- Use `#[default]` when possible (rust-lang/cargo#11197)
- Implement RFC 3289: source replacement ambiguity (rust-lang/cargo#10907)
- Use correct version of cargo in test (rust-lang/cargo#11193)
- Check empty input for login (rust-lang/cargo#11145)
- Add retry support to sparse registries (rust-lang/cargo#11069)
@ehuss ehuss mentioned this pull request Oct 12, 2022
4 of 11 tasks
@ehuss ehuss added this to the 1.66.0 milestone Nov 6, 2023
emooreatx added a commit to CIRISAI/CIRISEdge that referenced this pull request Jun 9, 2026
…es (partial #65)

PR #76's iOS PyO3 sim job failed with "no matching package named
\`heck\` found" — a transient index.crates.io resolution flake on
that specific runner. The non-sim iOS PyO3 job pulled the same dep
and succeeded within the same run; pure environmental flake, not
a code regression.

## Two-layer hardening (per web-search-validated best practices)

### Layer 1 — in-process retry budget bump

`CARGO_NET_RETRY=10` at workflow `env:` (default is 3). Cargo's
HTTP retry policy for sparse-registry index lookups uses this budget;
bumping from 3→10 catches the long-tail of transient flakes. Per
cargo-cooldown's COOLDOWN_HTTP_RETRIES precedent (default 2, max 8 —
we go one above for stateless CI). Costs nothing on the happy path.

### Layer 2 — cross-process retry on flake-prone lanes

`nick-fields/retry@v3` wraps the cargo invocations in the iOS /
Android cross-compile lanes (4 sites):

  - Android plain build (cargo ndk)
  - Android Chaquopy PyO3 build (cargo ndk with python-stub linkage)
  - iOS plain build (cargo build with -install_name baked in)
  - iOS PyO3 build (cargo build --target with PYO3_CROSS env)

3 attempts × 20 min each. Catches the case where cargo's in-process
retry budget is exhausted but a fresh process succeeds — observed on
PR #76's flake. linux-x86_64 gauntlet does NOT get the wrapper:
those lanes already cache via Swatinem/rust-cache + show no flake
history; adding retry-wrapper there is friction without benefit.

## Why not other options

- `cargo vendor` + `--offline`: ~500MB checkin + maintenance burden;
  overkill for our flake rate.
- Self-hosted crates.io mirror (Cloudsmith, panamax): warranted at
  team scale; overkill for a single-repo project.
- Commit `Cargo.lock`: legitimate option for binary distribution but
  separate concern from flake-hardening; deferred.

## Sources

- rust-lang/cargo#11069 — sparse-registry retry behavior
- https://cloudsmith.com/blog/cargo-sparse-registry — sparse protocol reliability tradeoffs
- https://crates.io/crates/cargo-cooldown — COOLDOWN_HTTP_RETRIES precedent
- https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol/ — sparse default since 1.70

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
emooreatx added a commit to CIRISAI/CIRISEdge that referenced this pull request Jun 9, 2026
…E_PROTOCOL_VERSION (partial #65) (#76)

* replication: scheduler glue + inbound channel (partial #65)

Closes the documented `Scheduler` non-goal in coordinator.rs / mod.rs.
The application can now wire one ReplicationScheduler over a fleet of
Initiator-side coordinators; the scheduler fires per-coordinator
tokio::time::interval ticks and drives each round through the
SendThenWait → recv inbound → DriveStep loop to completion (or
timeout). Responder rounds stay inline in the application's listen
loop (no separate scheduling needed — they're event-driven by inbound
Summary arrivals).

## Coordinator additions

- `ReplicationCoordinator::deliver_inbound(msg)` — push an inbound
  ReplicationMessage into the coordinator's internal queue. Called by
  the application's Transport::listen loop after parse_inbound_bytes
  recognizes a CRPL-framed message. Bounded mpsc(8), back-pressure
  surfaces as `NoRoundInProgress` on the listen-loop call (typically
  logged + dropped — the next interval tick recovers).
- `ReplicationCoordinator::recv_inbound()` — internal-facing; the
  scheduler awaits on this between SendThenWait phases.
- `INBOUND_CHANNEL_CAPACITY = 8` — one round in flight has at most 3
  messages queued (Summary + Diff + Deliver); 8 absorbs a slightly-
  late deliver while the scheduler is mid-step.

## scheduler.rs (new module)

- `SchedulerConfig` — cadence + round_timeout, default 30s + 10s
  (matches MISSION "near-realtime convergence" anchor).
- `ReplicationScheduler::new(config)` / `add_initiator(coord)` —
  builder; debug_assert that role is Initiator (Responders run inline
  in the listen loop, no scheduler).
- `run_until_cancelled(watch::Receiver<bool>)` — spawn one tokio task
  per registered coordinator, each with its own interval; observe the
  watch signal on every iteration, exit cleanly when it flips true.
- `run_with_events(cancel, Option<mpsc::Sender<(peer_id, RoundEvent)>>)`
  — same, plus per-round event surface for metrics consumers /
  τ_partial telemetry. Sink-closed is tolerated (round loops continue).
- `RoundEvent { Completed(RoundReport), Refused, TimedOut, Error(String) }`
  — what one round produced. The scheduler logs each via tracing
  spans regardless of whether an event_sink is wired.

## Cancellation model

`tokio::sync::watch::Receiver<bool>` — no new dep, lives in tokio core.
The caller holds the Sender, calls `.send(true)` to ask the scheduler
to shut down. Shutdown latency bounded by the slowest in-flight
round's round_timeout.

## Round timeout posture

If a SendThenWait phase's recv_inbound() blocks past round_timeout, the
scheduler emits RoundEvent::TimedOut, the coordinator's session
auto-resets via the existing Complete-path (the next interval tick
starts fresh). Anti-entropy is eventually-consistent by design — a
stuck round is a missed cadence, not a fault.

## Tests (5 new — 214 lib total, was 209)

- `add_initiator_tracks_len` — builder smoke.
- `default_config_pins_30s_and_10s` — defaults stable.
- `scheduler_drives_initiator_round_to_completion` — end-to-end across
  two real coordinators + in-memory transport pair + listen-loop-
  simulation tasks. Driver task feeds the scheduler's Alice; the
  responder-side Bob runs inline via a drain task that mirrors what
  the application's Transport::listen integration looks like.
  Asserts the round Completes with admitted=2 and InSync staleness —
  the full long-lived-session + scheduler + inbound-channel + transport
  stack is wired correctly.
- `scheduler_reports_timeout_when_peer_silent` — black-hole the
  outbound; assert RoundEvent::TimedOut surfaces.
- `scheduler_exits_on_cancel` — empty scheduler shuts down cleanly.

Uses real-time tokio (no `start_paused`) because the lib-test build
doesn't pull tokio's test-util feature; 10ms cadence + 500ms timeout
finishes each test in ~200ms.

## Verification

- cargo check clean across all 4 CI feature combos under -D warnings.
- cargo clippy clean on both CI clippy combos.
- cargo fmt clean.
- 214 lib tests pass (was 209; +5 scheduler).

## Replication ladder status

PRs #69 protocol → #70 coordinator → #71 directory adapters → #72
wire-frame prefix → #73 long-lived Session → this PR (scheduler glue).
Layer (c-2) production wiring (FederationDirectory blanket impl +
PyO3 init path) remains. Stacks on the long-lived-Session branch
because the scheduler depends on Session being held across calls.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* replication: document NAT-traversal posture (Reticulum mesh routing handles it)

Closes the documentation gap from PR #74's discussion. Edge does NOT
implement STUN / TURN / ICE — and shouldn't:

- Reticulum routes by cryptographic destination hash, NOT endpoint
  address. Transit nodes carry packets without decrypt capability,
  so the relay can't read what it's carrying.
- The federation topology supplies the public-side peer set by
  construction: registry servers (substrate, public by definition);
  CIRISLens (public observability surface — opt-in is policy-layer
  consent, not transit-layer reachability; relays carry what they
  can't read); agent 2.9.6 CEG/RET-native community-server-opt-in;
  any other CEG 0.15 community peer with a public interface.
- HTTP transport stays the one exception — its accept-side needs
  operator-supplied port-forward / reverse-proxy. Documented as
  such; operators behind hard NATs should use Reticulum (MISSION
  §1.4 canonical anyway).

Documentation-only — no code change.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* replication: v1 wire format — 9-variant EnvelopeKind + WIRE_PROTOCOL_VERSION (partial #65)

FSD/REPLICATION_WIRE_FORMAT_V1.md anchors v1 as the version-stable
replication contract. Locks the answers to the design questions
raised in #65: envelope_hash semantics, envelope_bytes wire format,
EnvelopeKind taxonomy, anti-entropy wire shape, version-byte advance
path to CIRIS 2.0.

## What's in this PR

### FSD/REPLICATION_WIRE_FORMAT_V1.md
- §3.1 envelope_hash = `original_content_hash` (LOCKED, CEG §0.9.2.1)
- §3.2 envelope_bytes = canonical(Signed*Record) via persist's
  PythonJsonCanonicalizer — edge does not define canonicalization
  (LOCKED, MISSION.md §3)
- §3.3 EnvelopeKind expanded 4→9 aligned 1:1 with persist's
  FederationDirectory put_* surface
- §3.4 Summary/Diff/Fetch/Deliver anti-entropy shape LOCKED
  (edge-defined per CIRISRegistry#58 "Edge owns propagation")
- §3.5 WIRE_PROTOCOL_VERSION byte added to frame preamble; v1=0x01,
  v2 reserved for CIRIS 2.0 (operational-data envelopes per #58 P2)
- §3.6 Layer (c-2) bridge shape (follow-up implementation)
- §3.7 PyO3 init shape (follow-up implementation)
- §5 Path to CIRIS 2.0 documented end-to-end

### Code: EnvelopeKind 4→9

Aligned with persist FederationDirectory:

  Key                              -> put_public_key
  Attestation                      -> put_attestation
  Revocation                       -> put_revocation
  IdentityOccurrence               -> put_identity_occurrence
  Family                           -> put_family
  Community                        -> put_community
  IdentityOccurrenceRevocation     -> put_identity_occurrence_revocation (v4.8.0)
  FamilyMembershipRevocation       -> put_family_membership_revocation (v4.8.0)
  CommunityMembershipRevocation    -> put_community_membership_revocation (v4.8.0)

Layer (c-2)'s apply_envelope_bytes dispatches via a simple match
on EnvelopeKind — no JSON shape sniffing.

### Code: WIRE_PROTOCOL_VERSION

5-byte preamble (was 4-byte magic):

  CRPL (4B) + VER (1B) + body...

WIRE_PROTOCOL_VERSION = 0x01 locks v1. `wrap_at_version` lets test
fixtures forge cross-version frames; `try_unwrap` surfaces typed
`ProtocolError::UnknownVersion(u8)` on a future v2 frame so a v1
receiver fails cleanly + scheduler reports the version mismatch.

### Code: strict v1 parser

`coordinator::parse_inbound_bytes` no longer tolerates bare-JSON
inputs (the pre-v1 rolling-upgrade tolerance from PR #70→#72). v1
LOCKS the wire format: every replication frame MUST carry the
CRPL+VER preamble. Bare-JSON inputs surface as
`CoordinatorError::Protocol(Decode)`. Wire-stable contract
going forward.

## Verification

- 218 lib tests pass (was 214 — added 4 wire-frame tests for the
  version byte path + 9-variant exhaustive round-trip)
- cargo clippy clean on both CI combos under -D warnings
- cargo fmt clean

## Path to 2.0 (FSD §5)

- **1.x line** — v1 wire (VER=0x01), 9 EnvelopeKinds, federation
  directory + R1/Q1 quorum-merge for trust data
- **2.0 cut** — Spock fully removed (CIRISRegistry#58 Phase 2).
  Operational-data envelopes defined by Registry. Wire bumps to
  VER=0x02 adding EnvelopeKind::Org/User/License/Partner. Edge
  tolerates both v1 and v2 during rolling-upgrade via the version
  byte at frame layer.

The wire-version byte (§3.5) is load-bearing — without it, the v1→v2
transition would require fleet-wide flag-day coordination. With it,
peers upgrade independently and the framing dispatch tells them what
they're receiving.

## Follow-ups (stacked on this PR)

- Layer (c-2) `FederationDirectoryReplicationBridge` — implements
  `ReplicationDirectory` over persist's `Arc<dyn FederationDirectory>`
  + cohort callback + hash→bytes cache; routes apply_envelope_bytes
  to the matching persist put_*.
- PyO3 init wiring — `replication_peers` config + runtime
  `PyEdge::register_replication_peer`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* ci: feature-combo gauntlet + proptest invariants for v1 wire format (partial #65)

Brings edge's CI to parity with persist's substrate-axis matrix
precedent (CIRISPersist v2.0.1 linux-x86_64-test). Edge's optional
features compose orthogonally; a regression in one transport lane
could hide behind another lane being enabled. The matrix catches
lane-isolation regressions at PR time, not at downstream consumer
integration.

## 5-entry feature-combo matrix on linux-x86_64-test

  - core:           transport-http only
  - reticulum:      transport-reticulum only (canonical wire / MISSION §1.4)
  - packet-radio:   transport-packet-radio only (CIRISEdge#53)
  - all-transports: transport-http + transport-reticulum + transport-packet-radio
  - pyo3-full:      transport-http + transport-reticulum + pyo3 (wheel canonical,
                    runs --all-targets so integration tests exercise too)

`fail-fast: false` per persist's pattern so a regression in one entry
doesn't mask regressions in others. Wall-clock = slowest single entry
(matrix is parallel), so the gauntlet adds runners but not wait time.
Per-combo `Swatinem/rust-cache` key prevents recompile thrash between
entries.

## Proptest properties for v1 wire format

New integration test tests/replication_wire_proptest.rs exercises 5
invariants over arbitrary inputs (256 cases per property by default):

  1. wrap/unwrap round-trip — any ReplicationMessage in, same out
  2. non-CRPL bytes always yield Ok(None) — never an error
  3. unknown version byte always surfaces typed UnknownVersion(v)
  4. EnvelopeKind JSON tag round-trips (all 9 variants)
  5. wrap == wrap_at_version(WIRE_PROTOCOL_VERSION) equivalence

Wall-clock: ~100ms for all 5 properties. Anchors the v1 wire-stable
contract from FSD §3 against arbitrary inputs — locks in PR time.

## Web-search-validated design

- Per `cargo-feature-combinations` (Rust-community canonical pattern):
    https://crates.io/crates/cargo-feature-combinations
- Per `doc.rust-lang.org/cargo/guide/continuous-integration.html`:
    https://doc.rust-lang.org/cargo/guide/continuous-integration.html
- Network-protocol CI fuzzing best practices (10-30 min budget for
  protocol fuzzing; we ship the proportionate proptest move at ~100ms,
  fuzzing follow-up if needed):
    https://link.springer.com/article/10.1007/s12243-024-01058-w

## What this does NOT add

- Full protocol fuzzing — disproportionate to current value; can be a
  follow-up under a time-boxed CI job if anti-entropy round shapes
  start tripping up downstream consumers.
- cargo-nextest — current tests run in <3s per combo; nextest's
  parallelism benefit doesn't materialize at this scale.
- Workspace-wide testing — edge is a single crate; no workspace.

Local verification:
  - All 5 matrix combos cargo test clean (186/180/196/218/218 tests)
  - All 4 lane-isolation combos cargo clippy clean under -D warnings
  - Both CI clippy combos clean
  - cargo fmt clean
  - 5/5 proptest properties pass in 100ms

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* release(v1.5.0): substrate bump persist v4.9.0 → v4.10.0 + 10th EnvelopeKind::LocationProof + re-enable ciris-verify pin-override

First version-stable replication line. Anchors the path to CIRIS 2.0
per FSD §5: v1 wire is locked from this version forward; protocol
breakages bump WIRE_PROTOCOL_VERSION.

## Substrate: persist v4.9.0 → v4.10.0

Persist v4.10.0 (#184 / #154) ships TWO things relevant to edge:

1. **CEG 0.8 §0.8.1 location_proof primitive** — V068 federation
   location_proofs table + LocationProof / SignedLocationProof types +
   put_location_proof on all 3 backends + pure-Rust h3o geospatial
   helpers (validate_location_cell, h3_cell_contained). H3 resolution
   ≤ 7 enforced at the substrate — a producer can't over-share
   precise location even if client UI gating fails.

2. **pyproject ciris-verify pin bumped <5 → >=5,<6** — closes the
   Python-side ceiling that blocked the v1.4.x conformance matrix
   from carrying verify==5.0.0 alongside persist 4.8.0/4.9.0. Edge's
   extract-substrate-pins workflow can now re-enable the
   ciris-verify pin-override extraction.

## 10th EnvelopeKind variant: LocationProof

Replication's EnvelopeKind taxonomy stays 1:1 with persist's
FederationDirectory put_* surface (FSD §3.3). Added the 10th variant
LocationProof (wire tag "location_proof") so the v1 wire is locked at
10 variants from this version forward. CEG-native location proofs
propagate via federation gossip on the same Summary/Diff/Deliver
shape as the other 9 kinds.

FSD §3.3 amended to the 10-variant table.

## ci.yml: re-enable ciris-verify pin-override extraction

The v1.4.0/v1.4.1 cycle disabled the ciris-verify pin-override
extraction because persist 4.8.0/4.9.0's pyproject still pinned
ciris-verify<5 (incompatible with edge's Cargo verify v5.0.0 pin).
Persist v4.10.0 fixes the pyproject ceiling, so the workflow
re-enables the override. The conformance matrix can now install
verify==5.0.0 alongside persist==4.10.0 cleanly.

Job docstring updated to reflect the history (v1.4 workaround) +
the rationale for re-enabling (v4.10.0 pyproject fix).

## Verification

- All 4 CI feature combos clean under RUSTFLAGS=-D warnings
- cargo clippy clean on both CI clippy combos
- cargo fmt clean
- 218 lib tests green; 5 proptest properties green (~100ms)
- 10-variant exhaustive round-trip test updated
- proptest generator covers all 10 variants

race_repro deferred to post-PyPI-propagation (persist 4.10.0 not
yet on PyPI as of commit time; will re-run conformance once propagated).

## Path to CIRIS 2.0 (FSD §5)

This is the first cut shipping the full v1 wire-stable shape:

- v1 wire format LOCKED (10-variant EnvelopeKind + WIRE_PROTOCOL_VERSION
  = 0x01 in frame preamble + strict CRPL+VER parser; no bare-JSON
  tolerance)
- Federation directory + R1/Q1 quorum-merge for trust + location
  data (v4.10.0)
- Anti-entropy protocol shape Summary/Diff/Fetch/Deliver
- Scheduler + listen-loop integration model documented
- Layer (c-2) bridge implementation remains as follow-up (PyO3 init
  + FederationDirectoryReplicationBridge)

The v2 cut (Spock removal per CIRISRegistry#58 Phase 2) will bump
WIRE_PROTOCOL_VERSION to 0x02 adding EnvelopeKind::Org/User/License/
Partner for operational-data envelopes. Peers upgrade peer-by-peer
via the version byte; no fleet-wide flag-day required.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* ci: harden cross-compile lanes against transient crates.io index flakes (partial #65)

PR #76's iOS PyO3 sim job failed with "no matching package named
\`heck\` found" — a transient index.crates.io resolution flake on
that specific runner. The non-sim iOS PyO3 job pulled the same dep
and succeeded within the same run; pure environmental flake, not
a code regression.

## Two-layer hardening (per web-search-validated best practices)

### Layer 1 — in-process retry budget bump

`CARGO_NET_RETRY=10` at workflow `env:` (default is 3). Cargo's
HTTP retry policy for sparse-registry index lookups uses this budget;
bumping from 3→10 catches the long-tail of transient flakes. Per
cargo-cooldown's COOLDOWN_HTTP_RETRIES precedent (default 2, max 8 —
we go one above for stateless CI). Costs nothing on the happy path.

### Layer 2 — cross-process retry on flake-prone lanes

`nick-fields/retry@v3` wraps the cargo invocations in the iOS /
Android cross-compile lanes (4 sites):

  - Android plain build (cargo ndk)
  - Android Chaquopy PyO3 build (cargo ndk with python-stub linkage)
  - iOS plain build (cargo build with -install_name baked in)
  - iOS PyO3 build (cargo build --target with PYO3_CROSS env)

3 attempts × 20 min each. Catches the case where cargo's in-process
retry budget is exhausted but a fresh process succeeds — observed on
PR #76's flake. linux-x86_64 gauntlet does NOT get the wrapper:
those lanes already cache via Swatinem/rust-cache + show no flake
history; adding retry-wrapper there is friction without benefit.

## Why not other options

- `cargo vendor` + `--offline`: ~500MB checkin + maintenance burden;
  overkill for our flake rate.
- Self-hosted crates.io mirror (Cloudsmith, panamax): warranted at
  team scale; overkill for a single-repo project.
- Commit `Cargo.lock`: legitimate option for binary distribution but
  separate concern from flake-hardening; deferred.

## Sources

- rust-lang/cargo#11069 — sparse-registry retry behavior
- https://cloudsmith.com/blog/cargo-sparse-registry — sparse protocol reliability tradeoffs
- https://crates.io/crates/cargo-cooldown — COOLDOWN_HTTP_RETRIES precedent
- https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol/ — sparse default since 1.70

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* ci: extend retry-wrapper to cargo bench --no-run (heck flake follow-up)

PR #76's CI hit the same `heck` index flake on the `cargo bench --no-run`
job that earlier hit iOS PyO3 sim. CARGO_NET_RETRY=10 didn't catch it
— consistent with the hypothesis that this is a sustained regional
crates.io index sync hiccup rather than transient network noise.

Wrap the bench step with nick-fields/retry@v3 (same shape as the
iOS/Android lanes added in the prior commit). Same 3 attempts × 15
min budget — generous since the bench is ~30s on a warm cache.

After this commit, every flake-prone lane that touches the crates.io
index has cross-process retry. linux-x86_64 gauntlet stays without
the wrapper because (a) it's never flaked, (b) the matrix already
caps wall-clock at slowest entry, (c) adding wrapper friction without
flake history is unwarranted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-sparse-registry Area: http sparse registries S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants