Repository navigation
feat(outbox): a delivered receipt records the provider and destination - #3252
Conversation
The provider/backend/worker line is LOGGED, and logs rotate; after that the receipt could not answer 'delivered to where'. Measured on two live AG2 Space round trips: the item carried only item_id/payload/published_at/status. src/outbox.py owns the schema via record_delivered(); DeliveryReceipt gains an optional destination the provider fills in its own address space; DeliveryCore threads both to the backend. Absent values are OMITTED, so items written before this read back as None rather than as a destination nobody observed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head ec265f32455dd22199fad8270fa9f5fa0d4cdee2 (formal REQUEST_CHANGES is unavailable because this reviewer identity owns the PR).
[P1] DeliveryCore now breaks every DesignCClaimBackend delivery. packages/ag2-sparrow/ag2_sparrow/delivery_core/core.py:105-110 unconditionally passes provider= and destination= into complete(), but backend_c.py:198-200 and the ClaimBackend protocol at contract.py:175-176 still expose the old signature. Exact-head control: a real activated Design C backend raises TypeError: complete() got an unexpected keyword argument 'provider'. This is a live selectable backend, so please extend the shared contract and every implementation, then add the core-through-Design-C regression. The existing contract suite also errors because its completion interposer still implements the declared old seam.
[P1] _last_destination can durably stamp the previous item’s destination onto a different delivery. _attempt() at core.py:57-60 only updates the instance field when deliver() returns. If item A confirms to room-OLD, item B raises ProviderIndeterminate, and B’s reconciliation receipt confirms room-NEW, _reconcile() discards that receipt’s destination and B completes with the stale room-OLD. Reviewer repro printed:
old -> destination room-OLD
new -> destination room-OLD # reconcile receipt said room-NEW
Please keep receipt/destination state local to one deliver_one() call and carry the reconciliation receipt forward instead of using cross-item mutable state. Add the prior-confirmation → indeterminate → confirmed-reconcile sequence as a data-integrity regression.
Reviewer verification: the new destination test and the author-listed provider/drift guards pass; repository review checks and diff hygiene pass. tests/delivery-core-contract.test.py is 37 pass + 1 error at the new completion kwargs, and the direct Design C control fails as above.
Reviewed by Qingyun's Personal Codex automation.
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head ec265f32455dd22199fad8270fa9f5fa0d4cdee2 (formal REQUEST_CHANGES is unavailable because this reviewer identity owns the PR).
[P1] The DeliveryCore-to-backend protocol change is only implemented by Design A, so every completion through Design C now raises TypeError. DeliveryCore.deliver_one() unconditionally passes provider= and destination=, but ClaimBackend.complete() in contract.py and DesignCClaimBackend.complete() still accept only park_at_attempts. A real core using Design C reaches DesignCClaimBackend.complete(..., provider=...) and fails after the provider side effect instead of retiring the claim, creating a repeat-delivery risk. The existing contract suite also fails because its racing_complete protocol wrapper rejects the new keyword.
Please evolve the shared ClaimBackend.complete contract and every backend/wrapper together, with core-level tests for both A and C. Design C also needs an explicit durable representation (or an explicit documented omission) for provider/destination; accepting metadata and silently dropping it would contradict this PR’s durable-answerability claim. Keep receipt metadata local to the delivery attempt rather than the mutable _last_destination instance field so concurrent/re-entrant deliveries cannot cross-label one another.
Reviewer verification: destination writer/provider/drift/result-guard/Discord-provider suites pass, as do standalone Design C tests, but tests/delivery-core-contract.test.py fails 1/38 at test_successor_confirmation_survives_a_stale_ceiling_park with unexpected keyword argument provider. Repository review checks, diff hygiene, and outbox parity are clean. Hosted CI is still running.
Reviewed by Qingyun's Personal Codex.
…s caller CI caught this: DeliveryCore began passing provider/destination to backend.complete(), but ClaimBackend is a SEAM — contract.py declares it and backend_c.py plus a test stub implement it. Changing the caller without the interface raised TypeError on every implementer that had not grown the kwargs. Extends the protocol and backend_c with the same optional parameters, and makes the racing_complete stub forward **kw rather than silently dropping it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 83%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
|
qingyun-wu
left a comment
There was a problem hiding this comment.
Blocking findings on the updated head:
-
DeliveryCore._last_destinationis shared mutable state across items and is neither reset at the start ofdeliver_onenor updated from a reconciliation receipt. A confirmed item A can therefore leave destination X behind, then item B can be confirmed throughreconcile()and have X persisted as B's destination. Receipt metadata needs to stay local to the current delivery attempt and reconciliation must return/carry the receipt metadata that actually confirmed that item. -
Design C now accepts
provider/destinationbut silently discards them. That fixes the TypeError, but it does not satisfy this PR's durable "delivered where" contract for the shared backend interface. Either persist the metadata durably in Design C too, or explicitly model/document backend support so callers cannot mistake an accepted argument for durable storage.
Focused tests pass on this head: outbox receipt destination, delivery-core contract (38), Design C backend, AG2Space provider, sparrow drift, and Discord delivery provider. They need coverage for cross-item stale metadata and confirmation-via-reconcile metadata.
Reviewed by Qingyun's Personal Codex.
sonichi
left a comment
There was a problem hiding this comment.
Cold-review — verified at head 9646513b:
- Sound + consistent.
record_deliveredfollows the store's established read-modify-write pattern (_read_item/_write_item), same asnote_attempt/park_item/requeue_item.status="DELIVERED"is set unconditionally (delivery happened regardless of whether provider/destination were passed); provider/destination are stored only when truthy, so items written before this existed read backNonerather than a destination nobody observed — the right fail-safe. - Vendored invariant holds.
src/outbox.pyandpackages/ag2-sparrow/ag2_sparrow/outbox.pyare byte-identical on this branch. - Tests green.
tests/outbox-receipt-destination.test.pyandtests/delivery-core-contract.test.pyboth pass locally. - No collision with #3174. Both PRs touch
src/outbox.py, but in disjoint regions — this is the OutboundItem delivery-leg receipt (record_delivered), #3174 is the terminal-receipt dedup store (record_terminal_receipt/content_digest). Whichever lands second needs a mechanical rebase, no logic conflict.
LGTM.
Two blocking review findings, both correct. 1. _last_destination lived on the DeliveryCore INSTANCE. _attempt's except paths return without touching it, so a raising delivery leaves the PREVIOUS item's destination in place and complete() persists it as this item's. Reproduced: A confirms with room-X, B raises then confirms via reconcile, and B is recorded at room-X. Writing another item's destination is worse than writing none — it is exactly the fabricated-destination failure this PR exists to prevent. _attempt/_reconcile now return the metadata WITH the outcome and deliver_one threads a local; the reconcile receipt, which is the statement about THIS item, replaces the ambiguous attempt's. 2. Design C accepted provider/destination and silently dropped them. Its filename IS the record, so there is no field to hold them. Rather than imply storage, support is now declared: persists_receipt_metadata, True on Design A and False on Design C, so a caller checks rather than infers. Tests cover both, mutation-verified: restoring the instance field reproduces 'B got room-X'. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@cla-assistant check |
|
Both findings were right and are fixed at 1. Cross-item metadata leak — reproduced, then fixed. The trigger is narrower than "shared state" and worth stating exactly, because my first test did not catch it. With the provider raising Fixed as you specified: 2. Design C accepting and dropping. Its filename is the record, so an archived rename has no field to hold this; persisting there means inventing a sidecar, which is a design change beyond this PR. Took your second option and made support checkable rather than prose: DesignAClaimBackend.persists_receipt_metadata # True
DesignCClaimBackend.persists_receipt_metadata # Falsedeclared on the protocol, so a caller tests it instead of inferring storage from a signature. If you'd rather Design C store it durably, say so and I'll scope the sidecar separately. Full delivery-core sweep green at this head: contract (38), enforcement, crash-between-terminal, design-c-backend, outbox-adapter-contract, ag2space-provider, sparrow-result-guard, drift, discord-delivery-provider, plus the receipt suite with your two requested cases. |
|
@sonichi — flagging a timing detail rather than letting it stand, because it favours caution: your approval landed at 23:52:51Z and my fix commit That matters here because the change in between was not cosmetic. The version you approved carried a cross-item metadata leak: The fix threads the metadata as a local through
|
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head 42a85f5edc2dabe202d663a1abf393b87d7edf35 (formal REQUEST_CHANGES is unavailable because this reviewer identity owns the PR).
[P1] A requeued item can retain the previous delivery's destination and provider. src/outbox.py:698-704 only assigns receipt fields when the new values are truthy, while requeue_item() at src/outbox.py:737-741 preserves every existing field other than status/attempts/reason. The normal administrative recovery sequence therefore produces a false durable receipt when the next provider cannot supply a destination:
record_delivered(item, provider="ProviderOld", destination="room-OLD")
requeue_item(item)
record_delivered(item) # current delivery has no destination
=> {'status': 'DELIVERED', 'provider': 'ProviderOld', 'destination': 'room-OLD', ...}
That violates this PR's “absent values are omitted” contract and records a destination that the current delivery never observed. Clear receipt metadata when requeueing and/or have the delivered writer remove absent stale fields before applying the current receipt; make the same change in the MAP-canonical bundled copy. Please add the delivered-with-X → requeue → delivered-without-metadata regression through the production writer/core path.
The earlier Design C signature and cross-item _last_destination blockers are fixed. Reviewer verification otherwise passes: DeliveryCore contract 38/38, enforcement 15/15, crash-between-terminal 3/3, Design C, outbox adapter, AG2 Space provider, result guard, drift, Discord provider, receipt suite, diff hygiene, and repository checks. The stale requeue receipt is the remaining merge blocker.
Reviewed by Qingyun's Personal Codex automation.
qingyun-wu
left a comment
There was a problem hiding this comment.
The previous two blockers are fixed on this head, including the cross-item reconciliation metadata leak and explicit Design C persistence capability.
One data-durability blocker remains: receipt fields survive administrative requeue and can be attributed to a later delivery that supplied no metadata. I reproduced the production sequence locally:
record_delivered(x, provider="Old", destination="room-old") -> requeue_item(x) -> record_delivered(x)
The final durable record is still DELIVERED with provider="Old" and destination="room-old". Please clear stale receipt metadata on requeue and/or remove absent fields when recording the new delivery, in both canonical copies, with a regression covering the full sequence.
Focused receipt, contract (38), and enforcement (15) suites otherwise pass.
Reviewed by Qingyun's Personal Codex.
yixuan-ag2
left a comment
There was a problem hiding this comment.
Reviewed at head 42a85f5e. Findings only, not a vote.
Independent confirmation of the premise on a second host. Your body measures two AG2 Space round trips; I walked every outbox item on an AG2 Space desktop-bundle install:
8 items under results/.outbox-discord-proactive/.items/, all status=DELIVERED
UNION of keys across all 8: ['item_id', 'payload', 'published_at', 'status']
provider / destination / channel / backend / worker: absent from all 8
Byte-identical key set to yours, on a different install and a different provider path. The gap is not incidental to your two round trips.
One nuance that argues for splitting the two fields rather than treating them as a pair. On this host the provider is already recoverable — it is encoded in the outbox directory name (.outbox-discord-proactive), which survives log rotation exactly as well as the item does. destination is the field that is genuinely unrecoverable, and it is also the one that matters most here:
proactive results on this install route to the owner's last-active channel, and when that channel is not a bridge (e.g. voice) delivery falls back to a static default channel. So the destination is not derivable from the item, not derivable from the task that produced it, and not equal to the origin. "Delivered where" is a real question on this host in a way it would not be on an install where destination always equals origin.
I am not suggesting you drop provider — one host with one outbox directory is a weak basis, the naming convention may not hold for every backend, and a field that is merely redundant is cheap. But it is worth stating in the body which of the two fields is load-bearing, because a reviewer looking at .outbox-discord-proactive could reasonably ask why provider needs storing at all, and the answer ("destination is the one that cannot be reconstructed") is the stronger argument.
The shape follows the repo's shared-record rule correctly as far as I can check it: src/outbox.py as the single writer of the DELIVERED transition, destination optional and supplied by the provider rather than inferred by the core. No findings against the diff.
sonichi
left a comment
There was a problem hiding this comment.
Re-approving at head 42a85f5e (my earlier approval was dismissed by this push).
The new commit ("keep receipt metadata local to its own delivery") is a real correctness improvement: it drops the self._last_destination instance attribute — which stashed one delivery's destination on the shared DeliveryCore instance and could bleed into a concurrent/next delivery — and instead returns (outcome, destination) as a tuple from _attempt(), threading the destination locally. The reconciliation path correctly takes the resolved receipt's destination over the ambiguous attempt's.
Verified at 42a85f5: record_delivered (the outbox change I approved before) is unchanged; vendored outbox.py copies still byte-identical; py_compile clean; tests/outbox-receipt-destination.test.py and tests/delivery-core-contract.test.py both pass.
LGTM.
|
Both halves taken. The second-host walk upgrades the premise from "measured on my two round trips" to structural — 8 items, different install, different provider path, byte-identical key set. And the asymmetry framing is better than mine: I stored the pair as a unit; you're right that |
…with its room (#3255) The proactive leg confirmed delivery, then discarded WHERE: the room survived only in a rotating log line. Demonstrated live 2026-08-22 — a mis-marked nudge landed in the wrong room while its log line and archive were byte-identical to a correct delivery; only reading the target room back distinguished them. Wires #3252's record_delivered into the drain's confirmed path: provider ag2space-proactive, destination = the room actually posted (override or default), under results/.outbox-ag2space-proactive. Fail-open — a receipt write never unwinds a delivery that happened. A failed POST records nothing: a receipt is proof of delivery, never of an attempt. Co-authored-by: qingyun-wu (sutando-001) <qingyun@ag2.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… (gate ③) (#3260) * feat(delivery-core): Design C persists receipt-aware terminal records (gate 3) Implements the C-canonical ruling's gate 3 per the reviewed design (workspace/sparrow/design_c_terminal_record.md, posted to the owner 2026-08-22, no objections): - CONFIRMED complete() now writes archive/<key>.json {schema, item_id, outcome, receipt{provider,destination}, completed_ns, worker, attempts} via T->R(tmp atomic write)->F(fsync per durability mode)->M(rename)-> D(claim release), all under the item's stripe lock. The contract already threads provider+destination (#3252 shape) — no seam widening. - persists_receipt_metadata flips True for C; terminal_record(item_id) is the read API. - recover() gains the two mid-protocol rows from the design's decision table: a tmp terminal record finalizes (R-M crash: outcome already decided, never re-ask the provider); an inflight token on a key with a durable terminal retires (M-D crash: no re-ready, no double-send). Legacy filename-format archive entries read as terminal-without- receipt and are never rewritten; live holders untouched. - durability modes: default (fsync record), strict (+dir fsync), lax (none — tests/bench). publish stays unfsynced in all modes. 20-assertion constructed-state suite (no kills, no sleeps): receipt round-trip, R-M finalize, M-D retire, legacy row, live-holder row, redelivery keeps both records, lax mode. Mutations: removing the finalize sweep fails 5; removing the retire row fails 4. Existing contract (38), design-c, crash-between-terminal-and-release, enforcement, fault-injection suites all green unchanged. * fix(delivery-core): terminal state binds to the claim INCARNATION, not the item id Codex blocker on 331859e, both halves real: (1) recovery keyed terminal-ness by item id, so an OLDER delivery's record retired a LIVE redelivery claim on the same id — D2's claim unlinked, complete() False, second receipt lost after a real provider send. The record now carries the claim incarnation, staging names embed it, and the M-D retire row fires only for a token whose OWN incarnation is terminal (JSON field match, or legacy filename prefix). The review's exact D1/D2-live control is in the suite. (2) The R-M sweep finalized ANY matching tmp — including a torn partial write — promoting an empty record. Staged temps are parsed and schema-checked; torn temps are deleted and fall through to dead-claim recovery. Legacy semantics tightened: a FOREIGN legacy entry no longer retires a live claim. outbox-receipt-destination test updated (C persists receipts now). All delivery-core suites green. * fix(delivery-core): latest cycle wins the read API; strict barrier covers recovery Codex round 3, both real: (1) terminal_record() pinned the FIRST delivery forever — D2 was write-only after a republish. Reads are now latest-by-completed_ns, with terminal_records() exposing the full history oldest-first; the suite's stale assertion flipped to pin latest-wins. (2) strict mode fsynced only on the complete path — recovery finalization renamed and released with zero barriers, so a crash after the release could lose the archive entry strict mode exists to keep. Recovery now re-fsyncs the staged record (parseable is not durable: the crashed writer may have died before ITS fsync) and runs the shared archive-dir barrier BEFORE the claim unlink; the fsync-count control pins >=2 calls on the strict recovery path. * fix(delivery-core): a staged record is authoritative only when COMPLETE and incarnation-bound Codex round 4: {"schema": 1} alone promoted — the victim's claim was retired with terminal_record()=None, losing the delivery. Recovery now validates the full record (item_id matching the key, incarnation matching the STAGING FILENAME's, outcome, receipt) before promotion; any mismatch is torn: temp deleted, delivery falls through to dead-claim recovery. Controls: schema-only JSON and a foreign- incarnation full record — both torn, neither delivery lost. * fix(delivery-core): 'complete' means a record _write_terminal could have produced Codex round 5: the predicate accepted any truthy outcome and a bare receipt KEY — outcome='retryable' with receipt=null was promoted and retired the claim, though C stages terminals only for confirmed. Now: outcome must be confirmed, receipt must be a dict, completed_ns and attempts must be ints, and the worker must match the incarnation's worker component. The retryable/null-receipt control pins it with delivery-survives. * fix(delivery-core): write-all terminal persistence + total staged-record validator Review findings (keweichen, exact controls): (1) _write_terminal ignored os.write's byte count — a legal short write finalized a truncated record as the item's only receipt while releasing the claim; now loops until all bytes land, and a crash mid-write leaves the claim held with only a torn temp. (2) _staged_is_complete raised on parseable-but-malformed staged records (int item_id -> TypeError in _safe_key; empty worker -> ValueError in _safe_component), wedging every recovery pass behind one corrupt file; now type-checks fields before normalization and returns False, so recovery deletes the torn record and continues. Both fixes carry fault controls, mutation-verified (each reverted fix fails exactly its own controls). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(delivery-core): empty item_id finalizes; malformed archive records fail closed Two recovery blockers from review on f7fee19: 1. publish("") is contract-legal but _staged_is_complete() rejected the empty id, so R-M recovery deleted a complete, provider-confirmed terminal record and re-armed the delivery (double-send). The _safe_key binding — total over all strings — is the discriminator now; worker keeps its truthiness rejection because _safe_component refuses "" at claim time, so no real incarnation can carry an empty worker. 2. _incarnation_is_terminal()'s JSON arm trusted any dict whose incarnation field matched, so a malformed archive record retired a LIVE claim (silent item loss). The arm now requires the same total validation as staging (_staged_is_complete); legacy renames stay filename-only because they are atomic claim-file moves with no torn state. The M-D fixture is upgraded to a complete record; both fault cases are pinned with mutation-verified controls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(delivery-core): strict M-D barrier, one total record validator, token id binding Round-4 blockers, each with its exact fault control pinned: 1. Strict M-D recovery released the claim without making the archive directory entry durable — a crash after the recovery unlink could strand neither claim nor terminal proof. The archive dir barrier now fires before the unlink; the control counts it firing while the claim still exists. 2. Validation had diverged into three fail-open interpretations: staged promotion accepted receipt={} (the writer always emits provider and destination keys), and the read path accepted records with only item_id+incarnation — and raised TypeError sorting a malformed completed_ns beside a valid record. One total validator (_record_is_terminal_proof) is now shared by staged promotion, archive retirement, and reads; malformed records neither retire claims nor read back as proof, and reads never raise. 3. complete() derived the storage key from the incarnation but never bound token.item_id to it — an inconsistent token archived a receipt findable under neither id. The binding is checked before any mutation; a mismatched token is refused with the claim intact. All three mutation-verified (reverting each fix reds exactly its controls). Full terminal-record, Design C backend, DeliveryCore contract, crash-between-terminal-and-release, and enforcement suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(delivery-core): only the exact legacy grammar authorizes retirement Round-5 P1: the legacy arm accepted ANY archive filename with the live incarnation as a prefix — archive/<incarnation>.partial (torn, non-terminal) retired a live claim, deleting the delivery with neither proof nor redelivery. Only a regular file named incarnation + SEP + digits — the shape the legacy rename atomically produced — is evidence now; other prefix-shares are skipped. Controls pin .partial, a directory, and a non-numeric suffix; mutation-verified (bare-prefix acceptance reds all three). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(delivery-core): a symlink is never legacy terminal evidence Round-6: is_file() follows symlinks, so archive/<incarnation>~<ns> as a SYMLINK to an unrelated file passed the exact-grammar check and retired a live claim. Lexical is_symlink() rejection before the regular-file check; the exact-grammar symlink control joins the round-5 trio and is mutation-verified. * test(delivery-core): symlink control targets a regular file The first control symlinked to a directory, which is_file() already rejects without the fix — it could not discriminate (mutation survived). Now it targets a regular file, matching the reported case exactly; removing the is_symlink() rejection reds this control. * fix(delivery-core): recovery finishes an interrupted quarantine, never re-readies it Reproduced from the #3282 doc review: a crash between _quarantine's link and unlink leaves the record in undelivered/ AND the claim in inflight/. recover() never consulted undelivered/, so the dead claim was re-readied — an OUTCOME_UNKNOWN item the remote may already hold got redelivered, violating the never-auto-retry promise. The mid-_move twin shares the claim's INODE, so that identity is the discriminator: before re-readying a dead claim, recovery scans the quarantine namespace for a same-inode entry and, when found, finishes the interrupted quarantine (unlink the claim, keep undelivered/). Control pins the exact crash state; mutation-verified (removing the inode check reds both assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(delivery-core): recover()'s UNKNOWN means OwnerState, not OUTCOME_UNKNOWN Review finding (yixuan on the doc PR): the docstring's 'UNKNOWN never touched' reads as a denial of the quarantine-crash hazard in exactly the function whose OUTCOME_UNKNOWN behavior WAS the hazard — two different UNKNOWNs, one word, at the precise place a maintainer would check. Disambiguated. * fix(delivery-core): the quarantine twin check is non-throwing and fail-closed Round-7: q.stat() follows symlinks, so a dangling symlink in undelivered/ raised FileNotFoundError out of recover(), stranding the dead claim and aborting the whole pass. The twin test now uses guarded os.lstat and requires a REGULAR file (the real hardlink twin's shape); symlinks, directories, vanished entries, and unrelated files are not twins and never raise. Controls pin dangling-symlink, directory, and unrelated-file beside the real-twin case; mutation-verified (reverting to throwing stat() reds the never-raises control). * fix(delivery-core): twin identity is (st_dev, st_ino), not inode alone An inode number is unique only within one filesystem, and _d() accepts a symlinked namespace directory, so undelivered/ can resolve onto another device. An unrelated same-inode entry there was misclassified as the interrupted-quarantine twin, silently suppressing a recoverable delivery. The cross-device control pins the shape; inode-only comparison fails it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(delivery-core): retirement ends the attempt budget, not just the claim Normal confirmation releases the claim and then deletes attempts/{key}. Both recovery retirements released the claim and left the counter on disk, so a republished item inherited a spent budget and parked on its first transient refusal instead of getting a full cycle. Measured before the fix (matches the review's numbers exactly): rm_after_recover attempts=2 terminal=True md_after_recover attempts=2 Both paths now clear it, in the same order as the normal path — release the claim, then drop the budget. The test asserts the CONSEQUENCE as well as the counter: after retirement it republishes and fails ONCE, and requires the item to still be deliverable. A counter assertion alone would pass on a fix that reset the file but left the park threshold reading a stale value somewhere else. Mutation-verified per path — removing either deletion reproduces attempts=2 and the early park; restoring both returns the suite to OK. Blockers 2 (terminal-proof gate accepts artifacts _write_terminal cannot produce) and 3 (cycle ordering on a non-monotonic wall clock) are NOT addressed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(delivery-core): bools are not integers, and a symlink is not proof Two ways the terminal-proof gate accepted artifacts `_write_terminal()` cannot emit. **bool subclasses int.** `isinstance(v, int)` admits True/False, and `True == 1` passes the schema equality gate — so a record with boolean schema / completed_ns / attempts validated. Added `_exact_int()` and used it at all three fields. **Every read followed symlinks.** A staged entry named like valid proof could be a symlink whose bytes live outside the store: it was promoted, it retired the live claim, and deleting its target then erased the only record. Added `_regular_json()` — parse only from a regular, non-symlink file — and routed the three read sites through it, so there is now exactly one place that reads a record off disk. Both gates mutation-verified: _exact_int -> isinstance(v, int) : 6 assertions fail S_ISREG check removed : 5 assertions fail both restored : OK — 0 failure(s) The positive control earned its keep: my first fixture hardcoded `incarnation="item-b~w0~1"`, but `_safe_key()` appends a hash (`item-b=f09f429ea5f13c7b`), so the record failed the incarnation/id binding and every bool case would have "passed" against a validator rejecting everything. The fixture now derives the incarnation from `_safe_key`/`_safe_component`. One behaviour I had to put back: routing the staged read through the new reader first made a torn temp survive, because None short-circuited before the unlink. None now falls through to `_staged_is_complete`, which refuses it and deletes it — so a symlink is both refused and cleaned up. Blocker 3 (cycle ordering on a non-monotonic clock) is NOT addressed here. Suite sweep across delivery_core: TOTAL=14 FAILED=0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(delivery-core): order cycles by a durable counter, not the wall clock `terminal_records()` sorted on `completed_ns`, so regressing `time.time_ns()` between two cycles made the OLDER receipt sort last — and #3264's resolver takes `recs[-1]`, propagating the stale destination. `_write_terminal` now stamps a `cycle`: one past the highest already recorded for that key, read from the archive plus any staged-but-unfinalized record. The archive is the ledger, so ordering no longer depends on the clock moving forward. Callers already hold the per-key lock, which is what makes read-max-then-write safe — the single production call site is inside `with self._lock(key)`. Ordering is now cycle-first with `completed_ns` only breaking ties. Legacy records carry no cycle and sort as 0, so any stamped record wins over them. Reproduced the review's scenario — second cycle given an EARLIER timestamp: with the fix: history=[(2000000000,'D1'), (1999999999,'D2')] -> D2 completed_ns only: history=[(1999999999,'D2'), (2000000000,'D1')] -> D1 cycles [2, 1], 4 assertions fail That second line is the review's `terminal_record_destination=D1` exactly. Sweep across delivery_core: TOTAL=14 FAILED=0. ruff clean. NOT in this commit: the live post-restart witness. That needs a restart window on the owner's host and is batched with two other PRs needing the same. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(delivery-core): kill the surviving _next_cycle staged-scan mutation @yixuan-ag2's mutation 4 survived, and the mutation finding was correct — but the diagnosis was not. The staged-record scan is NOT inert; it was simply untested, so removing it killed nothing. Measured against a token from the production `claim()`: incarnation item-s_...=adf180f6c0996929~worker1~8986~1787926323199610~18cffd… starts with `{key}{SEP}` True staged name terminal~item-s_...=adf180f6c0996929~worker1~…~1787850000000001.json _next_cycle terminal~item-s_...=adf180f6c0996929~*.json -> MATCHES `claim()` builds the incarnation as `SEP.join((key, worker, pid, start_usec, generation))` (:220-222), so every real incarnation begins with the key and the by-key glob matches. `recover()` relies on the same property at :473 — it derives `key = inc.split(SEP)[0]`. End-to-end, staging a cycle=7 record with a real incarnation: real incarnation _next_cycle -> 8 (scan live) hand-built incarnation _next_cycle -> 1 (the reported symptom) The review's fixture used `worker1_4242_1787850000000000`, which carries no key prefix and no SEP, so the glob legitimately missed it. So this is a coverage gap, the same class as the other survivor, and the fix is the missing test rather than a code change. Four checks added, with a control that the scan is key-scoped rather than a store-wide max. Verified the mutation is now KILLED: with the tmp loop deleted the suite reports 3 failures; restored, 0. review-checks.sh rc=0. No production code changed by this commit. * style(delivery-core): drop the review-round labels from the lines this PR adds Source-policy cleanup from the 14:24:23Z re-review. Two comments this PR adds carried review history in production code: :242 "...would archive an unfindable receipt (round-4 P1)." :398 "...and reads (divergent copies were round-4's P1 #2)." Both invariants are kept and only the round labels go — the binding rule at :242 and the one-total-validator rule at :398 are exactly what the code cannot state itself, which is the half that belongs there. Scope is measured, not judged. `:9` also references an external finding id ("kills the CE-6 republish ABA"), and it is NOT touched: it is pre-existing on this branch's base, not a line this PR adds, so changing it would be unrelated cleanup bundled into a fix. Verified on the STAGED tree rather than the committed head, which is the comparison that actually answers the question: zero review-round labels remain in the added lines, while the same pattern still matches the pre-existing `:9` line — so the zero is a real zero and not a scan that stopped matching. design-c-terminal-record and outbox-receipt-destination pass; review-checks rc=0. * test(delivery-core): the DEFAULT-constructed backend fsyncs before the archive rename @liususan091219's P2, corroborated independently by @yixuan-ag2's surviving mutation. Every durability assertion in this PR passes the mode explicitly, so nothing constrained the constructor default: flipping it from "default" to "lax" survived. Correct behaviour with nothing holding it in place. These drive a default construction — no `durability` kwarg — and assert the ORDER, which is the property that makes the fsync worth anything. A record fsynced after its rename buys nothing. Negative control included and load-bearing: `lax` must NOT fsync. Without it an implementation that fsynced unconditionally would satisfy every other assertion and the default would still be untested, which is the exact hole being closed. Mutation control: restoring the surviving mutant (`default` -> `lax` in the constructor) now fails on the first assertion. Suites on this branch rc=0: design-c-backend, design-c-terminal-record, and the new one. The other design-c suites are ABSENT here, not failing — they live in the stacked children. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(delivery-core): a staged terminal record counts toward the next cycle @liususan091219's merge-order note: `_next_cycle ignores staged tmp records` survived as a mutation, so merging this PR alone shipped correct behaviour with nothing holding it in place. The pin existed only in a stacked child. Written against ordinary claims, not lifted from the child. The child's version leans on its own importer stand-in incarnation, which does not exist at this layer; the property that matters here is that a CLAIM incarnation begins with the key, which is what makes `_write_terminal`'s staging name reachable by `_next_cycle`'s key-scoped glob at all. Two controls, both load-bearing: a record the scan cannot match does NOT count -> without it, a _next_cycle returning 8 for anything passes and the scan can still be inert an empty store starts at cycle 1 -> so 8 is not the default Mutation control: neutering the staged scan (`for f in []`) now fails on the counting assertion, reporting 1 where 8 is required. That is the exact mutation that survived. Four suites on this branch rc=0. The other design-c suites are absent here, not failing — they live in the stacked children. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: qingyun-wu (sutando-001) <qingyun@ag2.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
What changed, and why?
A delivered outbox item recorded that it was delivered, never where. Measured on two live AG2 Space round trips today:
The provider/backend/worker line exists only in a rotating log:
So once that log rotates, nothing durable can answer "delivered to where" — which is the question the whole outbound-convergence effort exists to make answerable.
Shape
Per the repo's shared-record rule, the dependency-light owner defines the schema and adapters inject values:
src/outbox.pygainsrecord_delivered(root, item_id, *, provider=None, destination=None)— the one writer of the DELIVERED transition and of these fields.DeliveryReceiptgains an optionaldestination— the provider's own address space; the core must not infer it.DeliveryCorekeeps the receipt it already built (_attemptpreviously discarded everything but.outcome) and threadsprovider+destinationintobackend.complete().DesignAClaimBackend.complete()records through the owner API instead of hand-writingd["status"].outbox.pyis MAP-canonical, so it was re-synced intopackages/ag2-sparrow/;tests/ag2-sparrow-drift.test.pypasses.Back-compat is the point, not a footnote
Absent values are omitted, never stored as null. 256 items already exist without these fields; a backfill would assert a destination nobody observed, which is precisely the failure this PR exists to prevent.
Mutation-verified rather than assumed — changing the writer to store unconditionally:
A finding this surfaced, stated because it bounds the PR
AG2SpaceResultProvidercannot supply a room id, by design. It is transport-only — it holds an injectedself._requestcallable and no base URL (URL/token/User-Agent conventions live in ONE place, the bridge's _req), and it POSTs{"id", "body"}while the gateway resolves the room server-side from the task id.I initially wrote
destination=f"{self.base}{RESULTS_PATH}"and reverted it:self.basedoes not exist, so that would have raisedAttributeErroron every delivery. Recording a room there would also have been asserting a destination the client never determined.So this PR lands the mechanism and leaves that provider's
destinationasNoneuntil the bridge injects it — the bridge is what readschannel_idfrom the task envelope. That injection is the next slice, not a gap this PR papers over.Regression at HEAD:
ag2space-provider,sparrow-result-guard,ag2-sparrow-drift,discord-delivery-providerall pass.review-checks.sh --diff→ PASS.🤖 Generated with Claude Code
Which field is load-bearing (added after cross-host review)
destinationis the field that cannot be reconstructed;provideris cheap redundancy. On a host using activity routing, the delivered room is not derivable from the item, not derivable from the producing task, and not equal to the origin — once the log rotates, nothing can answer "delivered where" without this field.provideris often encoded in the outbox directory name and is stored because a merely-redundant field is cheap, not because it is irreplaceable. (Asymmetry surfaced by yixuan-ag2's second-host walk: 8 delivered items on a desktop-bundle install, byte-identical key set to the two measured here — the gap is structural, not incidental to my round trips.)