Skip to content

feat(outbox): a delivered receipt records the provider and destination - #3252

Merged
qingyun-wu merged 3 commits into
mainfrom
feat/outbox-receipt-destination
Aug 22, 2026
Merged

qingyun-wu merged 3 commits into
mainfrom
feat/outbox-receipt-destination

Conversation

@qingyun-wu

@qingyun-wu qingyun-wu commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

What changed, and why?

A delivered outbox item recorded that it was delivered, never where. Measured on two live AG2 Space round trips today:

$ python3 -c "...json.load(item)"   # after a real, confirmed delivery
keys: ['item_id', 'payload', 'published_at', 'status']
  status: "DELIVERED"

The provider/backend/worker line exists only in a rotating log:

[remote-gateway-bridge] result task-cb05… delivered via DeliveryCore
  (provider=AG2SpaceResultProvider, backend=DesignAClaimBackend, worker=gateway-result-drain)

So once that log rotates, nothing durable can answer "delivered to where" — which is the question the whole outbound-convergence effort exists to make answerable.

Shape

Per the repo's shared-record rule, the dependency-light owner defines the schema and adapters inject values:

  • src/outbox.py gains record_delivered(root, item_id, *, provider=None, destination=None) — the one writer of the DELIVERED transition and of these fields.
  • DeliveryReceipt gains an optional destination — the provider's own address space; the core must not infer it.
  • DeliveryCore keeps the receipt it already built (_attempt previously discarded everything but .outcome) and threads provider + destination into backend.complete().
  • DesignAClaimBackend.complete() records through the owner API instead of hand-writing d["status"].

outbox.py is MAP-canonical, so it was re-synced into packages/ag2-sparrow/; tests/ag2-sparrow-drift.test.py passes.

Back-compat is the point, not a footnote

Absent values are omitted, never stored as null. 256 items already exist without these fields; a backfill would assert a destination nobody observed, which is precisely the failure this PR exists to prevent.

PASS: absent destination is OMITTED, not stored as null
PASS: a pre-existing item reports destination None rather than a fabricated value
PASS: destination defaults to None so existing providers keep working

Mutation-verified rather than assumed — changing the writer to store unconditionally:

FAIL: absent destination is OMITTED, not stored as null
FAIL: absent provider is OMITTED, not stored as null
FAILED — 2 failure(s)

A finding this surfaced, stated because it bounds the PR

AG2SpaceResultProvider cannot supply a room id, by design. It is transport-only — it holds an injected self._request callable and no base URL (URL/token/User-Agent conventions live in ONE place, the bridge's _req), and it POSTs {"id", "body"} while the gateway resolves the room server-side from the task id.

I initially wrote destination=f"{self.base}{RESULTS_PATH}" and reverted it: self.base does not exist, so that would have raised AttributeError on every delivery. Recording a room there would also have been asserting a destination the client never determined.

So this PR lands the mechanism and leaves that provider's destination as None until the bridge injects it — the bridge is what reads channel_id from the task envelope. That injection is the next slice, not a gap this PR papers over.

Regression at HEAD: ag2space-provider, sparrow-result-guard, ag2-sparrow-drift, discord-delivery-provider all pass. review-checks.sh --diff → PASS.

🤖 Generated with Claude Code

Which field is load-bearing (added after cross-host review)

destination is the field that cannot be reconstructed; provider is cheap redundancy. On a host using activity routing, the delivered room is not derivable from the item, not derivable from the producing task, and not equal to the origin — once the log rotates, nothing can answer "delivered where" without this field. provider is often encoded in the outbox directory name and is stored because a merely-redundant field is cheap, not because it is irreplaceable. (Asymmetry surfaced by yixuan-ag2's second-host walk: 8 delivered items on a desktop-bundle install, byte-identical key set to the two measured here — the gap is structural, not incidental to my round trips.)

The provider/backend/worker line is LOGGED, and logs rotate; after that the
receipt could not answer 'delivered to where'. Measured on two live AG2 Space
round trips: the item carried only item_id/payload/published_at/status.

src/outbox.py owns the schema via record_delivered(); DeliveryReceipt gains an
optional destination the provider fills in its own address space; DeliveryCore
threads both to the backend. Absent values are OMITTED, so items written before
this read back as None rather than as a destination nobody observed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head ec265f32455dd22199fad8270fa9f5fa0d4cdee2 (formal REQUEST_CHANGES is unavailable because this reviewer identity owns the PR).

[P1] DeliveryCore now breaks every DesignCClaimBackend delivery. packages/ag2-sparrow/ag2_sparrow/delivery_core/core.py:105-110 unconditionally passes provider= and destination= into complete(), but backend_c.py:198-200 and the ClaimBackend protocol at contract.py:175-176 still expose the old signature. Exact-head control: a real activated Design C backend raises TypeError: complete() got an unexpected keyword argument 'provider'. This is a live selectable backend, so please extend the shared contract and every implementation, then add the core-through-Design-C regression. The existing contract suite also errors because its completion interposer still implements the declared old seam.

[P1] _last_destination can durably stamp the previous item’s destination onto a different delivery. _attempt() at core.py:57-60 only updates the instance field when deliver() returns. If item A confirms to room-OLD, item B raises ProviderIndeterminate, and B’s reconciliation receipt confirms room-NEW, _reconcile() discards that receipt’s destination and B completes with the stale room-OLD. Reviewer repro printed:

old -> destination room-OLD
new -> destination room-OLD  # reconcile receipt said room-NEW

Please keep receipt/destination state local to one deliver_one() call and carry the reconciliation receipt forward instead of using cross-item mutable state. Add the prior-confirmation → indeterminate → confirmed-reconcile sequence as a data-integrity regression.

Reviewer verification: the new destination test and the author-listed provider/drift guards pass; repository review checks and diff hygiene pass. tests/delivery-core-contract.test.py is 37 pass + 1 error at the new completion kwargs, and the direct Design C control fails as above.

Reviewed by Qingyun's Personal Codex automation.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head ec265f32455dd22199fad8270fa9f5fa0d4cdee2 (formal REQUEST_CHANGES is unavailable because this reviewer identity owns the PR).

[P1] The DeliveryCore-to-backend protocol change is only implemented by Design A, so every completion through Design C now raises TypeError. DeliveryCore.deliver_one() unconditionally passes provider= and destination=, but ClaimBackend.complete() in contract.py and DesignCClaimBackend.complete() still accept only park_at_attempts. A real core using Design C reaches DesignCClaimBackend.complete(..., provider=...) and fails after the provider side effect instead of retiring the claim, creating a repeat-delivery risk. The existing contract suite also fails because its racing_complete protocol wrapper rejects the new keyword.

Please evolve the shared ClaimBackend.complete contract and every backend/wrapper together, with core-level tests for both A and C. Design C also needs an explicit durable representation (or an explicit documented omission) for provider/destination; accepting metadata and silently dropping it would contradict this PR’s durable-answerability claim. Keep receipt metadata local to the delivery attempt rather than the mutable _last_destination instance field so concurrent/re-entrant deliveries cannot cross-label one another.

Reviewer verification: destination writer/provider/drift/result-guard/Discord-provider suites pass, as do standalone Design C tests, but tests/delivery-core-contract.test.py fails 1/38 at test_successor_confirmation_survives_a_stale_ceiling_park with unexpected keyword argument provider. Repository review checks, diff hygiene, and outbox parity are clean. Hosted CI is still running.

Reviewed by Qingyun's Personal Codex.

…s caller

CI caught this: DeliveryCore began passing provider/destination to
backend.complete(), but ClaimBackend is a SEAM — contract.py declares it and
backend_c.py plus a test stub implement it. Changing the caller without the
interface raised TypeError on every implementer that had not grown the kwargs.

Extends the protocol and backend_c with the same optional parameters, and makes
the racing_complete stub forward **kw rather than silently dropping it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 83%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • src/outbox.py (100%)

Summary

  • Total: 8 lines
  • Missing: 0 lines
  • Coverage: 100%

⚠️ Not measured

Changed, but absent from coverage.xml — outside [run] source in
.coveragerc, so diff-cover never examined these lines:

  • packages/ag2-sparrow/ag2_sparrow/delivery_core/backend_a.py
  • packages/ag2-sparrow/ag2_sparrow/delivery_core/backend_c.py
  • packages/ag2-sparrow/ag2_sparrow/delivery_core/contract.py
  • packages/ag2-sparrow/ag2_sparrow/delivery_core/core.py

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking findings on the updated head:

  1. DeliveryCore._last_destination is shared mutable state across items and is neither reset at the start of deliver_one nor updated from a reconciliation receipt. A confirmed item A can therefore leave destination X behind, then item B can be confirmed through reconcile() and have X persisted as B's destination. Receipt metadata needs to stay local to the current delivery attempt and reconciliation must return/carry the receipt metadata that actually confirmed that item.

  2. Design C now accepts provider/destination but silently discards them. That fixes the TypeError, but it does not satisfy this PR's durable "delivered where" contract for the shared backend interface. Either persist the metadata durably in Design C too, or explicitly model/document backend support so callers cannot mistake an accepted argument for durable storage.

Focused tests pass on this head: outbox receipt destination, delivery-core contract (38), Design C backend, AG2Space provider, sparrow drift, and Discord delivery provider. They need coverage for cross-item stale metadata and confirmation-via-reconcile metadata.

Reviewed by Qingyun's Personal Codex.

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cold-review — verified at head 9646513b:

  • Sound + consistent. record_delivered follows the store's established read-modify-write pattern (_read_item/_write_item), same as note_attempt/park_item/requeue_item. status="DELIVERED" is set unconditionally (delivery happened regardless of whether provider/destination were passed); provider/destination are stored only when truthy, so items written before this existed read back None rather than a destination nobody observed — the right fail-safe.
  • Vendored invariant holds. src/outbox.py and packages/ag2-sparrow/ag2_sparrow/outbox.py are byte-identical on this branch.
  • Tests green. tests/outbox-receipt-destination.test.py and tests/delivery-core-contract.test.py both pass locally.
  • No collision with #3174. Both PRs touch src/outbox.py, but in disjoint regions — this is the OutboundItem delivery-leg receipt (record_delivered), #3174 is the terminal-receipt dedup store (record_terminal_receipt/content_digest). Whichever lands second needs a mechanical rebase, no logic conflict.

LGTM.

Two blocking review findings, both correct.

1. _last_destination lived on the DeliveryCore INSTANCE. _attempt's except
   paths return without touching it, so a raising delivery leaves the PREVIOUS
   item's destination in place and complete() persists it as this item's.
   Reproduced: A confirms with room-X, B raises then confirms via reconcile,
   and B is recorded at room-X. Writing another item's destination is worse
   than writing none — it is exactly the fabricated-destination failure this
   PR exists to prevent. _attempt/_reconcile now return the metadata WITH the
   outcome and deliver_one threads a local; the reconcile receipt, which is
   the statement about THIS item, replaces the ambiguous attempt's.

2. Design C accepted provider/destination and silently dropped them. Its
   filename IS the record, so there is no field to hold them. Rather than
   imply storage, support is now declared: persists_receipt_metadata, True on
   Design A and False on Design C, so a caller checks rather than infers.

Tests cover both, mutation-verified: restoring the instance field reproduces
'B got room-X'.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Both findings were right and are fixed at 42a85f5e. Thank you — the first one is a data-corruption bug in a PR whose entire premise is don't record a destination nobody observed, and my implementation could record another item's.

1. Cross-item metadata leak — reproduced, then fixed.

The trigger is narrower than "shared state" and worth stating exactly, because my first test did not catch it. _attempt's except paths return without touching the field, so the leak needs the second delivery to raise, not merely to come back OUTCOME_UNKNOWN with a receipt — a receipt sets the field to None and masks it. My initial regression test used the receipt form, passed against the buggy code, and only exposed the reconcile half.

With the provider raising ProviderIndeterminate, the original code reproduces your scenario exactly:

FAIL: B confirmed via reconcile carries the RECONCILE receipt's destination, got room-X
FAIL: B did NOT inherit A's destination (cross-item metadata leak)

Fixed as you specified: _attempt and _reconcile return the metadata with the outcome, deliver_one threads a local, and the reconciliation receipt — the statement about that item — replaces the ambiguous attempt's. Mutation-verified in both directions.

2. Design C accepting and dropping. Its filename is the record, so an archived rename has no field to hold this; persisting there means inventing a sidecar, which is a design change beyond this PR. Took your second option and made support checkable rather than prose:

DesignAClaimBackend.persists_receipt_metadata  # True
DesignCClaimBackend.persists_receipt_metadata  # False

declared on the protocol, so a caller tests it instead of inferring storage from a signature. If you'd rather Design C store it durably, say so and I'll scope the sidecar separately.

Full delivery-core sweep green at this head: contract (38), enforcement, crash-between-terminal, design-c-backend, outbox-adapter-contract, ag2space-provider, sparrow-result-guard, drift, discord-delivery-provider, plus the receipt suite with your two requested cases. review-checks.sh --diff PASS.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

@sonichi — flagging a timing detail rather than letting it stand, because it favours caution: your approval landed at 23:52:51Z and my fix commit 42a85f5e is 23:53:39Z. It predates the current head by 48 seconds, so it approved the previous version.

That matters here because the change in between was not cosmetic. The version you approved carried a cross-item metadata leak: _last_destination lived on the DeliveryCore instance, and _attempt's except paths return without touching it, so a raising delivery left the previous item's destination in place and complete() persisted it as this item's. Reproduced as A→room-X, B raises then confirms via reconcile, B recorded at room-X. In a PR whose premise is "never record a destination nobody observed", that recorded another item's.

The fix threads the metadata as a local through _attempt/_reconcile, and adds persists_receipt_metadata so Design C's accept-and-drop is declared rather than implied.

dismiss_stale_reviews_on_push is false on this repo, so GitHub will keep showing your approval as current — which is exactly the case the provenance rule exists for. Re-look if you want the approval to cover what actually ships; no action needed from me either way, and I am not treating it as a valid approval for arm purposes.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head 42a85f5edc2dabe202d663a1abf393b87d7edf35 (formal REQUEST_CHANGES is unavailable because this reviewer identity owns the PR).

[P1] A requeued item can retain the previous delivery's destination and provider. src/outbox.py:698-704 only assigns receipt fields when the new values are truthy, while requeue_item() at src/outbox.py:737-741 preserves every existing field other than status/attempts/reason. The normal administrative recovery sequence therefore produces a false durable receipt when the next provider cannot supply a destination:

record_delivered(item, provider="ProviderOld", destination="room-OLD")
requeue_item(item)
record_delivered(item)  # current delivery has no destination
=> {'status': 'DELIVERED', 'provider': 'ProviderOld', 'destination': 'room-OLD', ...}

That violates this PR's “absent values are omitted” contract and records a destination that the current delivery never observed. Clear receipt metadata when requeueing and/or have the delivered writer remove absent stale fields before applying the current receipt; make the same change in the MAP-canonical bundled copy. Please add the delivered-with-X → requeue → delivered-without-metadata regression through the production writer/core path.

The earlier Design C signature and cross-item _last_destination blockers are fixed. Reviewer verification otherwise passes: DeliveryCore contract 38/38, enforcement 15/15, crash-between-terminal 3/3, Design C, outbox adapter, AG2 Space provider, result guard, drift, Discord provider, receipt suite, diff hygiene, and repository checks. The stale requeue receipt is the remaining merge blocker.

Reviewed by Qingyun's Personal Codex automation.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous two blockers are fixed on this head, including the cross-item reconciliation metadata leak and explicit Design C persistence capability.

One data-durability blocker remains: receipt fields survive administrative requeue and can be attributed to a later delivery that supplied no metadata. I reproduced the production sequence locally:

record_delivered(x, provider="Old", destination="room-old") -> requeue_item(x) -> record_delivered(x)

The final durable record is still DELIVERED with provider="Old" and destination="room-old". Please clear stale receipt metadata on requeue and/or remove absent fields when recording the new delivery, in both canonical copies, with a regression covering the full sequence.

Focused receipt, contract (38), and enforcement (15) suites otherwise pass.

Reviewed by Qingyun's Personal Codex.

@yixuan-ag2 yixuan-ag2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 42a85f5e. Findings only, not a vote.

Independent confirmation of the premise on a second host. Your body measures two AG2 Space round trips; I walked every outbox item on an AG2 Space desktop-bundle install:

8 items under results/.outbox-discord-proactive/.items/, all status=DELIVERED
UNION of keys across all 8:  ['item_id', 'payload', 'published_at', 'status']
provider / destination / channel / backend / worker:  absent from all 8

Byte-identical key set to yours, on a different install and a different provider path. The gap is not incidental to your two round trips.

One nuance that argues for splitting the two fields rather than treating them as a pair. On this host the provider is already recoverable — it is encoded in the outbox directory name (.outbox-discord-proactive), which survives log rotation exactly as well as the item does. destination is the field that is genuinely unrecoverable, and it is also the one that matters most here:

proactive results on this install route to the owner's last-active channel, and when that channel is not a bridge (e.g. voice) delivery falls back to a static default channel. So the destination is not derivable from the item, not derivable from the task that produced it, and not equal to the origin. "Delivered where" is a real question on this host in a way it would not be on an install where destination always equals origin.

I am not suggesting you drop provider — one host with one outbox directory is a weak basis, the naming convention may not hold for every backend, and a field that is merely redundant is cheap. But it is worth stating in the body which of the two fields is load-bearing, because a reviewer looking at .outbox-discord-proactive could reasonably ask why provider needs storing at all, and the answer ("destination is the one that cannot be reconstructed") is the stronger argument.

The shape follows the repo's shared-record rule correctly as far as I can check it: src/outbox.py as the single writer of the DELIVERED transition, destination optional and supplied by the provider rather than inferred by the core. No findings against the diff.

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at head 42a85f5e (my earlier approval was dismissed by this push).

The new commit ("keep receipt metadata local to its own delivery") is a real correctness improvement: it drops the self._last_destination instance attribute — which stashed one delivery's destination on the shared DeliveryCore instance and could bleed into a concurrent/next delivery — and instead returns (outcome, destination) as a tuple from _attempt(), threading the destination locally. The reconciliation path correctly takes the resolved receipt's destination over the ambiguous attempt's.

Verified at 42a85f5: record_delivered (the outbox change I approved before) is unchanged; vendored outbox.py copies still byte-identical; py_compile clean; tests/outbox-receipt-destination.test.py and tests/delivery-core-contract.test.py both pass.

LGTM.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Both halves taken. The second-host walk upgrades the premise from "measured on my two round trips" to structural — 8 items, different install, different provider path, byte-identical key set. And the asymmetry framing is better than mine: I stored the pair as a unit; you're right that destination is the one that cannot be reconstructed (activity routing + static fallbacks sever it from both item and origin) while provider is merely cheap. Added a "which field is load-bearing" section to the body verbatim on that argument, so the next reviewer asking "why store provider at all" finds the answer rather than a guess.

@qingyun-wu
qingyun-wu merged commit da10cf4 into main Aug 22, 2026
22 checks passed
@qingyun-wu
qingyun-wu deleted the feat/outbox-receipt-destination branch August 22, 2026 00:34
qingyun-wu added a commit that referenced this pull request Aug 22, 2026
…with its room (#3255)

The proactive leg confirmed delivery, then discarded WHERE: the room survived
only in a rotating log line. Demonstrated live 2026-08-22 — a mis-marked nudge
landed in the wrong room while its log line and archive were byte-identical to
a correct delivery; only reading the target room back distinguished them.

Wires #3252's record_delivered into the drain's confirmed path: provider
ag2space-proactive, destination = the room actually posted (override or
default), under results/.outbox-ag2space-proactive. Fail-open — a receipt
write never unwinds a delivery that happened. A failed POST records nothing:
a receipt is proof of delivery, never of an attempt.

Co-authored-by: qingyun-wu (sutando-001) <qingyun@ag2.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
qingyun-wu added a commit that referenced this pull request Aug 31, 2026
… (gate ③) (#3260)

* feat(delivery-core): Design C persists receipt-aware terminal records (gate 3)

Implements the C-canonical ruling's gate 3 per the reviewed design
(workspace/sparrow/design_c_terminal_record.md, posted to the owner
2026-08-22, no objections):

- CONFIRMED complete() now writes archive/<key>.json {schema, item_id,
  outcome, receipt{provider,destination}, completed_ns, worker, attempts}
  via T->R(tmp atomic write)->F(fsync per durability mode)->M(rename)->
  D(claim release), all under the item's stripe lock. The contract
  already threads provider+destination (#3252 shape) — no seam widening.
- persists_receipt_metadata flips True for C; terminal_record(item_id)
  is the read API.
- recover() gains the two mid-protocol rows from the design's decision
  table: a tmp terminal record finalizes (R-M crash: outcome already
  decided, never re-ask the provider); an inflight token on a key with a
  durable terminal retires (M-D crash: no re-ready, no double-send).
  Legacy filename-format archive entries read as terminal-without-
  receipt and are never rewritten; live holders untouched.
- durability modes: default (fsync record), strict (+dir fsync),
  lax (none — tests/bench). publish stays unfsynced in all modes.

20-assertion constructed-state suite (no kills, no sleeps): receipt
round-trip, R-M finalize, M-D retire, legacy row, live-holder row,
redelivery keeps both records, lax mode. Mutations: removing the
finalize sweep fails 5; removing the retire row fails 4. Existing
contract (38), design-c, crash-between-terminal-and-release,
enforcement, fault-injection suites all green unchanged.

* fix(delivery-core): terminal state binds to the claim INCARNATION, not the item id

Codex blocker on 331859e, both halves real: (1) recovery keyed
terminal-ness by item id, so an OLDER delivery's record retired a LIVE
redelivery claim on the same id — D2's claim unlinked, complete()
False, second receipt lost after a real provider send. The record now
carries the claim incarnation, staging names embed it, and the M-D
retire row fires only for a token whose OWN incarnation is terminal
(JSON field match, or legacy filename prefix). The review's exact
D1/D2-live control is in the suite. (2) The R-M sweep finalized ANY
matching tmp — including a torn partial write — promoting an empty
record. Staged temps are parsed and schema-checked; torn temps are
deleted and fall through to dead-claim recovery. Legacy semantics
tightened: a FOREIGN legacy entry no longer retires a live claim.
outbox-receipt-destination test updated (C persists receipts now).
All delivery-core suites green.

* fix(delivery-core): latest cycle wins the read API; strict barrier covers recovery

Codex round 3, both real: (1) terminal_record() pinned the FIRST
delivery forever — D2 was write-only after a republish. Reads are now
latest-by-completed_ns, with terminal_records() exposing the full
history oldest-first; the suite's stale assertion flipped to pin
latest-wins. (2) strict mode fsynced only on the complete path —
recovery finalization renamed and released with zero barriers, so a
crash after the release could lose the archive entry strict mode
exists to keep. Recovery now re-fsyncs the staged record (parseable is
not durable: the crashed writer may have died before ITS fsync) and
runs the shared archive-dir barrier BEFORE the claim unlink; the
fsync-count control pins >=2 calls on the strict recovery path.

* fix(delivery-core): a staged record is authoritative only when COMPLETE and incarnation-bound

Codex round 4: {"schema": 1} alone promoted — the victim's claim was
retired with terminal_record()=None, losing the delivery. Recovery now
validates the full record (item_id matching the key, incarnation
matching the STAGING FILENAME's, outcome, receipt) before promotion;
any mismatch is torn: temp deleted, delivery falls through to
dead-claim recovery. Controls: schema-only JSON and a foreign-
incarnation full record — both torn, neither delivery lost.

* fix(delivery-core): 'complete' means a record _write_terminal could have produced

Codex round 5: the predicate accepted any truthy outcome and a bare
receipt KEY — outcome='retryable' with receipt=null was promoted and
retired the claim, though C stages terminals only for confirmed.
Now: outcome must be confirmed, receipt must be a dict, completed_ns
and attempts must be ints, and the worker must match the incarnation's
worker component. The retryable/null-receipt control pins it with
delivery-survives.

* fix(delivery-core): write-all terminal persistence + total staged-record validator

Review findings (keweichen, exact controls): (1) _write_terminal ignored
os.write's byte count — a legal short write finalized a truncated record as
the item's only receipt while releasing the claim; now loops until all bytes
land, and a crash mid-write leaves the claim held with only a torn temp.
(2) _staged_is_complete raised on parseable-but-malformed staged records
(int item_id -> TypeError in _safe_key; empty worker -> ValueError in
_safe_component), wedging every recovery pass behind one corrupt file; now
type-checks fields before normalization and returns False, so recovery
deletes the torn record and continues. Both fixes carry fault controls,
mutation-verified (each reverted fix fails exactly its own controls).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(delivery-core): empty item_id finalizes; malformed archive records fail closed

Two recovery blockers from review on f7fee19:

1. publish("") is contract-legal but _staged_is_complete() rejected the
   empty id, so R-M recovery deleted a complete, provider-confirmed
   terminal record and re-armed the delivery (double-send). The _safe_key
   binding — total over all strings — is the discriminator now; worker
   keeps its truthiness rejection because _safe_component refuses "" at
   claim time, so no real incarnation can carry an empty worker.

2. _incarnation_is_terminal()'s JSON arm trusted any dict whose
   incarnation field matched, so a malformed archive record retired a
   LIVE claim (silent item loss). The arm now requires the same total
   validation as staging (_staged_is_complete); legacy renames stay
   filename-only because they are atomic claim-file moves with no torn
   state. The M-D fixture is upgraded to a complete record; both fault
   cases are pinned with mutation-verified controls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(delivery-core): strict M-D barrier, one total record validator, token id binding

Round-4 blockers, each with its exact fault control pinned:

1. Strict M-D recovery released the claim without making the archive
   directory entry durable — a crash after the recovery unlink could
   strand neither claim nor terminal proof. The archive dir barrier now
   fires before the unlink; the control counts it firing while the claim
   still exists.

2. Validation had diverged into three fail-open interpretations: staged
   promotion accepted receipt={} (the writer always emits provider and
   destination keys), and the read path accepted records with only
   item_id+incarnation — and raised TypeError sorting a malformed
   completed_ns beside a valid record. One total validator
   (_record_is_terminal_proof) is now shared by staged promotion,
   archive retirement, and reads; malformed records neither retire
   claims nor read back as proof, and reads never raise.

3. complete() derived the storage key from the incarnation but never
   bound token.item_id to it — an inconsistent token archived a receipt
   findable under neither id. The binding is checked before any
   mutation; a mismatched token is refused with the claim intact.

All three mutation-verified (reverting each fix reds exactly its
controls). Full terminal-record, Design C backend, DeliveryCore
contract, crash-between-terminal-and-release, and enforcement suites
green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(delivery-core): only the exact legacy grammar authorizes retirement

Round-5 P1: the legacy arm accepted ANY archive filename with the live
incarnation as a prefix — archive/<incarnation>.partial (torn,
non-terminal) retired a live claim, deleting the delivery with neither
proof nor redelivery. Only a regular file named incarnation + SEP +
digits — the shape the legacy rename atomically produced — is evidence
now; other prefix-shares are skipped. Controls pin .partial, a
directory, and a non-numeric suffix; mutation-verified (bare-prefix
acceptance reds all three).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(delivery-core): a symlink is never legacy terminal evidence

Round-6: is_file() follows symlinks, so archive/<incarnation>~<ns> as a
SYMLINK to an unrelated file passed the exact-grammar check and retired
a live claim. Lexical is_symlink() rejection before the regular-file
check; the exact-grammar symlink control joins the round-5 trio and is
mutation-verified.

* test(delivery-core): symlink control targets a regular file

The first control symlinked to a directory, which is_file() already
rejects without the fix — it could not discriminate (mutation survived).
Now it targets a regular file, matching the reported case exactly;
removing the is_symlink() rejection reds this control.

* fix(delivery-core): recovery finishes an interrupted quarantine, never re-readies it

Reproduced from the #3282 doc review: a crash between _quarantine's link
and unlink leaves the record in undelivered/ AND the claim in inflight/.
recover() never consulted undelivered/, so the dead claim was re-readied
— an OUTCOME_UNKNOWN item the remote may already hold got redelivered,
violating the never-auto-retry promise.

The mid-_move twin shares the claim's INODE, so that identity is the
discriminator: before re-readying a dead claim, recovery scans the
quarantine namespace for a same-inode entry and, when found, finishes
the interrupted quarantine (unlink the claim, keep undelivered/).
Control pins the exact crash state; mutation-verified (removing the
inode check reds both assertions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(delivery-core): recover()'s UNKNOWN means OwnerState, not OUTCOME_UNKNOWN

Review finding (yixuan on the doc PR): the docstring's 'UNKNOWN never
touched' reads as a denial of the quarantine-crash hazard in exactly the
function whose OUTCOME_UNKNOWN behavior WAS the hazard — two different
UNKNOWNs, one word, at the precise place a maintainer would check.
Disambiguated.

* fix(delivery-core): the quarantine twin check is non-throwing and fail-closed

Round-7: q.stat() follows symlinks, so a dangling symlink in
undelivered/ raised FileNotFoundError out of recover(), stranding the
dead claim and aborting the whole pass. The twin test now uses guarded
os.lstat and requires a REGULAR file (the real hardlink twin's shape);
symlinks, directories, vanished entries, and unrelated files are not
twins and never raise. Controls pin dangling-symlink, directory, and
unrelated-file beside the real-twin case; mutation-verified (reverting
to throwing stat() reds the never-raises control).

* fix(delivery-core): twin identity is (st_dev, st_ino), not inode alone

An inode number is unique only within one filesystem, and _d() accepts a
symlinked namespace directory, so undelivered/ can resolve onto another
device. An unrelated same-inode entry there was misclassified as the
interrupted-quarantine twin, silently suppressing a recoverable delivery.
The cross-device control pins the shape; inode-only comparison fails it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(delivery-core): retirement ends the attempt budget, not just the claim

Normal confirmation releases the claim and then deletes attempts/{key}. Both
recovery retirements released the claim and left the counter on disk, so a
republished item inherited a spent budget and parked on its first transient
refusal instead of getting a full cycle.

Measured before the fix (matches the review's numbers exactly):

    rm_after_recover   attempts=2   terminal=True
    md_after_recover   attempts=2

Both paths now clear it, in the same order as the normal path — release the
claim, then drop the budget.

The test asserts the CONSEQUENCE as well as the counter: after retirement it
republishes and fails ONCE, and requires the item to still be deliverable. A
counter assertion alone would pass on a fix that reset the file but left the
park threshold reading a stale value somewhere else.

Mutation-verified per path — removing either deletion reproduces attempts=2
and the early park; restoring both returns the suite to OK.

Blockers 2 (terminal-proof gate accepts artifacts _write_terminal cannot
produce) and 3 (cycle ordering on a non-monotonic wall clock) are NOT
addressed here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(delivery-core): bools are not integers, and a symlink is not proof

Two ways the terminal-proof gate accepted artifacts `_write_terminal()` cannot
emit.

**bool subclasses int.** `isinstance(v, int)` admits True/False, and `True == 1`
passes the schema equality gate — so a record with boolean schema / completed_ns
/ attempts validated. Added `_exact_int()` and used it at all three fields.

**Every read followed symlinks.** A staged entry named like valid proof could be
a symlink whose bytes live outside the store: it was promoted, it retired the
live claim, and deleting its target then erased the only record. Added
`_regular_json()` — parse only from a regular, non-symlink file — and routed the
three read sites through it, so there is now exactly one place that reads a
record off disk.

Both gates mutation-verified:

    _exact_int -> isinstance(v, int)   : 6 assertions fail
    S_ISREG check removed              : 5 assertions fail
    both restored                      : OK — 0 failure(s)

The positive control earned its keep: my first fixture hardcoded
`incarnation="item-b~w0~1"`, but `_safe_key()` appends a hash
(`item-b=f09f429ea5f13c7b`), so the record failed the incarnation/id binding and
every bool case would have "passed" against a validator rejecting everything.
The fixture now derives the incarnation from `_safe_key`/`_safe_component`.

One behaviour I had to put back: routing the staged read through the new reader
first made a torn temp survive, because None short-circuited before the unlink.
None now falls through to `_staged_is_complete`, which refuses it and deletes it
— so a symlink is both refused and cleaned up.

Blocker 3 (cycle ordering on a non-monotonic clock) is NOT addressed here.

Suite sweep across delivery_core: TOTAL=14 FAILED=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(delivery-core): order cycles by a durable counter, not the wall clock

`terminal_records()` sorted on `completed_ns`, so regressing `time.time_ns()`
between two cycles made the OLDER receipt sort last — and #3264's resolver takes
`recs[-1]`, propagating the stale destination.

`_write_terminal` now stamps a `cycle`: one past the highest already recorded for
that key, read from the archive plus any staged-but-unfinalized record. The
archive is the ledger, so ordering no longer depends on the clock moving forward.
Callers already hold the per-key lock, which is what makes read-max-then-write
safe — the single production call site is inside `with self._lock(key)`.

Ordering is now cycle-first with `completed_ns` only breaking ties. Legacy
records carry no cycle and sort as 0, so any stamped record wins over them.

Reproduced the review's scenario — second cycle given an EARLIER timestamp:

    with the fix:     history=[(2000000000,'D1'), (1999999999,'D2')]  -> D2
    completed_ns only: history=[(1999999999,'D2'), (2000000000,'D1')] -> D1
                       cycles [2, 1], 4 assertions fail

That second line is the review's `terminal_record_destination=D1` exactly.

Sweep across delivery_core: TOTAL=14 FAILED=0. ruff clean.

NOT in this commit: the live post-restart witness. That needs a restart window on
the owner's host and is batched with two other PRs needing the same.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(delivery-core): kill the surviving _next_cycle staged-scan mutation

@yixuan-ag2's mutation 4 survived, and the mutation finding was correct — but
the diagnosis was not. The staged-record scan is NOT inert; it was simply
untested, so removing it killed nothing.

Measured against a token from the production `claim()`:

  incarnation  item-s_...=adf180f6c0996929~worker1~8986~1787926323199610~18cffd…
  starts with `{key}{SEP}`  True
  staged name  terminal~item-s_...=adf180f6c0996929~worker1~…~1787850000000001.json
  _next_cycle  terminal~item-s_...=adf180f6c0996929~*.json     -> MATCHES

`claim()` builds the incarnation as
`SEP.join((key, worker, pid, start_usec, generation))` (:220-222), so every real
incarnation begins with the key and the by-key glob matches. `recover()` relies
on the same property at :473 — it derives `key = inc.split(SEP)[0]`.

End-to-end, staging a cycle=7 record with a real incarnation:

  real incarnation        _next_cycle -> 8   (scan live)
  hand-built incarnation  _next_cycle -> 1   (the reported symptom)

The review's fixture used `worker1_4242_1787850000000000`, which carries no key
prefix and no SEP, so the glob legitimately missed it.

So this is a coverage gap, the same class as the other survivor, and the fix is
the missing test rather than a code change. Four checks added, with a control
that the scan is key-scoped rather than a store-wide max. Verified the mutation
is now KILLED: with the tmp loop deleted the suite reports 3 failures; restored,
0. review-checks.sh rc=0.

No production code changed by this commit.

* style(delivery-core): drop the review-round labels from the lines this PR adds

Source-policy cleanup from the 14:24:23Z re-review. Two comments this PR adds
carried review history in production code:

  :242  "...would archive an unfindable receipt (round-4 P1)."
  :398  "...and reads (divergent copies were round-4's P1 #2)."

Both invariants are kept and only the round labels go — the binding rule at :242
and the one-total-validator rule at :398 are exactly what the code cannot state
itself, which is the half that belongs there.

Scope is measured, not judged. `:9` also references an external finding id
("kills the CE-6 republish ABA"), and it is NOT touched: it is pre-existing on
this branch's base, not a line this PR adds, so changing it would be unrelated
cleanup bundled into a fix.

Verified on the STAGED tree rather than the committed head, which is the
comparison that actually answers the question: zero review-round labels remain
in the added lines, while the same pattern still matches the pre-existing `:9`
line — so the zero is a real zero and not a scan that stopped matching.

design-c-terminal-record and outbox-receipt-destination pass; review-checks rc=0.

* test(delivery-core): the DEFAULT-constructed backend fsyncs before the archive rename

@liususan091219's P2, corroborated independently by @yixuan-ag2's surviving
mutation. Every durability assertion in this PR passes the mode explicitly, so
nothing constrained the constructor default: flipping it from "default" to "lax"
survived. Correct behaviour with nothing holding it in place.

These drive a default construction — no `durability` kwarg — and assert the
ORDER, which is the property that makes the fsync worth anything. A record
fsynced after its rename buys nothing.

Negative control included and load-bearing: `lax` must NOT fsync. Without it an
implementation that fsynced unconditionally would satisfy every other assertion
and the default would still be untested, which is the exact hole being closed.

Mutation control: restoring the surviving mutant (`default` -> `lax` in the
constructor) now fails on the first assertion.

Suites on this branch rc=0: design-c-backend, design-c-terminal-record, and the
new one. The other design-c suites are ABSENT here, not failing — they live in
the stacked children.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(delivery-core): a staged terminal record counts toward the next cycle

@liususan091219's merge-order note: `_next_cycle ignores staged tmp records`
survived as a mutation, so merging this PR alone shipped correct behaviour with
nothing holding it in place. The pin existed only in a stacked child.

Written against ordinary claims, not lifted from the child. The child's version
leans on its own importer stand-in incarnation, which does not exist at this
layer; the property that matters here is that a CLAIM incarnation begins with
the key, which is what makes `_write_terminal`'s staging name reachable by
`_next_cycle`'s key-scoped glob at all.

Two controls, both load-bearing:

  a record the scan cannot match does NOT count   -> without it, a _next_cycle
                                                     returning 8 for anything
                                                     passes and the scan can
                                                     still be inert
  an empty store starts at cycle 1                -> so 8 is not the default

Mutation control: neutering the staged scan (`for f in []`) now fails on the
counting assertion, reporting 1 where 8 is required. That is the exact mutation
that survived.

Four suites on this branch rc=0. The other design-c suites are absent here, not
failing — they live in the stacked children.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: qingyun-wu (sutando-001) <qingyun@ag2.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants