Skip to content

Persist Claim delivery evidence for restart-safe verification - #76718

Merged
elefthei merged 8 commits into
mainfrom
fix/wq/durable-delivery-evidence
Nov 11, 2026
Merged

elefthei merged 8 commits into
mainfrom
fix/wq/durable-delivery-evidence

Conversation

@elefthei

@elefthei elefthei commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

The reported ledger contained 13 Work items that expired as DeliveryFailure(verification_exhausted) because delivery receipts lived in process-local WeakMaps and maintenance had no verifier. This change publishes bounded, authenticated DeliveryEvidence after the worker handler pass so maintenance can verify delivery by fresh native readback after process exit, without repeating writes or rewriting historical terminal decisions.

Review map

flowchart LR
  W["Worker safe_outputs<br/>handler pass"] -- "B3 serializable receipts" --> P["collect + publish<br/>(B4)"]
  P -- "B1 DeliveryEvidence op" --> L[("checked ledger")]
  L -. "B2 checkpoint keeps it<br/>while barrier pending" .-> L
  M["Scheduled maintenance<br/>(B5)"] -- "reads evidence" --> L
  M -- "fresh native readback" --> G["GitHub resources"]
  M -- "Result or DeliveryFailure" --> L
Loading
Behavior Commit Production files Tests
B1 — Immutable, bounded, worker-only ledger operation + generated contract 42c42d9e99 actions/setup/js/work_queue_policy.cjs
actions/setup/js/work_queue_replay.cjs
pkg/workqueue/delivery_evidence.go
pkg/workqueue/policy.go
pkg/workqueue/replay.go
pkg/workqueue/request.go
pkg/workqueue/schema/DeliveryEvidenceEffect.json
pkg/workqueue/schema/DeliveryEvidenceExpected.json
pkg/workqueue/schema/DeliveryEvidenceNoWriteOutput.json
pkg/workqueue/schema/DeliveryEvidenceOperation.json
pkg/workqueue/schema/DeliveryEvidenceOutput.json
pkg/workqueue/schema/DeliveryEvidenceWriteOutput.json
pkg/workqueue/schema/OperationsParameters.json
pkg/workqueue/schema/QueueCommit.json
pkg/workqueue/schema/QueueOperation.json
pkg/workqueue/schema/QueueRequest.json
pkg/workqueue/schema/RequestParameters.json
pkg/workqueue/types.go
specs/work-queue/transactions.tsp
actions/setup/js/work_queue_replay.test.cjs
pkg/workqueue/delivery_evidence_test.go
pkg/workqueue/policy_permitted_test.go
B2 — Pending evidence survives checkpoints and isolated projection clones dfcd0ea467 pkg/workqueue/checkpoint.go
pkg/workqueue/selection.go
pkg/workqueue/delivery_evidence_parity_test.go
B3 — Shared receipt selection/serialization and fresh explicit-receipt verification eb1902c9de actions/setup/js/upload_artifact.cjs
actions/setup/js/work_queue_claim_adapters.cjs
actions/setup/js/work_queue_claim_scope.cjs
actions/setup/js/work_queue_code_coverage.cjs
actions/setup/js/work_queue_code_scanning.cjs
actions/setup/js/work_queue_control_receipts.cjs
actions/setup/js/work_queue_delivery.cjs
actions/setup/js/work_queue_git_tree_adapter.cjs
actions/setup/js/work_queue_graphql_adapter.cjs
actions/setup/js/work_queue_native_creation_delivery.cjs
actions/setup/js/work_queue_rest_adapter.cjs
actions/setup/js/work_queue_upload_assets.cjs
actions/setup/js/upload_artifact_work_queue.test.cjs
actions/setup/js/work_queue_claim_adapters.test.cjs
actions/setup/js/work_queue_claim_scope.test.cjs
actions/setup/js/work_queue_control_receipts.test.cjs
actions/setup/js/work_queue_delivery.test.cjs
actions/setup/js/work_queue_git_tree_adapter_checks.cjs
actions/setup/js/work_queue_graphql_adapter_checks.cjs
actions/setup/js/work_queue_native_creation_delivery.test.cjs
actions/setup/js/work_queue_rest_adapter.test.cjs
actions/setup/js/work_queue_specialized_delivery.test.cjs
B4 — One complete evidence inventory after the worker handler pass b0a9fe0b99 actions/setup/js/safe_output_handler_manager.cjs
actions/setup/js/work_queue_delivery_evidence.cjs
actions/setup/js/work_queue_delivery_evidence.test.cjs
B5 — Authenticated maintenance readback and Result/DeliveryFailure settlement f7e1b5fecb actions/setup/js/finish_work_queue_claim.cjs
actions/setup/js/reconcile_work_queue_run.cjs
actions/setup/js/work_queue_delivery_recovery.cjs
actions/setup/js/work_queue_reconciler.cjs
actions/setup/js/work_queue_delivery_evidence.test_helpers.cjs
actions/setup/js/work_queue_delivery_evidence_adapters_restart.test.cjs
actions/setup/js/work_queue_delivery_evidence_native_restart.test.cjs
actions/setup/js/work_queue_delivery_evidence_restart.test.cjs
B6 — Specification, reference, and ADR contract 0b55078c94 docs/adr/76718-durable-claim-delivery-evidence.md
docs/src/content/docs/reference/work-queue.md
docs/src/content/docs/specs/work-queue-specification.md
—

| CI — Patched Go 1.27.4 baseline for GO-2026-6859 | adbf649e16 | .changeset/compiler-transport-trust.md
.devcontainer/devcontainer.json
.devcontainer/upgrade-tools.sh
.github/workflows/daily-code-debt-aider.lock.yml
.github/workflows/daily-code-debt-aider.md
.github/workflows/daily-go-test-stubs-aider.lock.yml
.github/workflows/daily-go-test-stubs-aider.md
.github/workflows/go.mod
.github/workflows/smoke-atomic.lock.yml
.github/workflows/smoke-atomic.md
.github/workflows/smoke-claude.lock.yml
.github/workflows/smoke-claude.md
.github/workflows/smoke-codex.lock.yml
.github/workflows/smoke-codex.md
.github/workflows/smoke-copilot-aoai-apikey.lock.yml
.github/workflows/smoke-copilot-aoai-apikey.md
.github/workflows/smoke-copilot-aoai-entra.lock.yml
.github/workflows/smoke-copilot-aoai-entra.md
.github/workflows/smoke-copilot-arm.lock.yml
.github/workflows/smoke-copilot-arm.md
.github/workflows/smoke-copilot.lock.yml
.github/workflows/smoke-copilot.md
.github/workflows/smoke-cursor.lock.yml
.github/workflows/smoke-cursor.md
.github/workflows/smoke-factory.lock.yml
.github/workflows/smoke-factory.md
.github/workflows/smoke-goose.lock.yml
.github/workflows/smoke-goose.md
.github/workflows/smoke-kiro.lock.yml
.github/workflows/smoke-kiro.md
.github/workflows/smoke-test-tools.lock.yml
.github/workflows/smoke-test-tools.md
.golangci.yml
CHANGELOG.md
go.mod
pkg/constants/version_constants.go
scripts/check-go-toolchain.sh
specs/security-architecture-spec.md | pkg/workflow/aider_discovery_test.go
pkg/workflow/runtime_go_baseline_test.go (removed source-only assertions)
pkg/workflow/runtime_setup_test.go
scripts/check-go-toolchain_test.sh |

B6 docs: specification §7.6 and §7.15, work-queue reference, ADR-76718.

Review notes

  • Evidence is input to fresh native readback, never proof of delivery.
  • 32 KiB / 128 outputs / 256 effects / 8 KiB adapter bounds are protocol constants, not Policy limits; B1 includes the previously missing TypeSpec source and reproducible generated schemas.
  • Failed or unsupported passes publish nothing: no partial inventory.
  • Maintenance gets one ≤15 s readback when it first sees the terminal run after the deadline.
  • Checkpoints omit the evidence key when empty, preserving historical digests.
  • The two pre-existing checkpoint functions retain nolint:largefunc rather than a lint-driven file split.

Verification

Refactored and rebuilt tips: 25 JS files / 769 tests passed; full Go workqueue package passed, including manager-restart parity and schema reproducibility. All six DeliveryEvidence commits passed go vet ./pkg/workqueue and loading their changed production CJS modules. The six-commit rebuilt tree matched the refactored snapshot exactly. The CI repair adds 42 files; all 103 changed files are attributed once above.

Restart proof: worker evidence survives cleared CJS modules; maintenance returns result with one total write. Tampered readback yields one DeliveryFailure and zero Results. The four evidence suites pass 26 tests; the moved receipt-rebinding test passes in claim-scope. Old and regenerated evidence schemas agree on 123 boundary cases.

CI repair: Go 1.27.4 fixes GO-2026-6859 / CVE-2026-97054. The exact failing scan now reports 0 reachable vulnerabilities. Release-floor tests reject 1.27.3 and accept 1.27.4; runtime selection/root tests and all 365 workflow compilations pass. All 14 affected generated locks select the patched version. Source-only/default-version echo assertions were removed rather than re-pinned; executed release-floor and runtime behavior checks remain. make agent-report-progress-no-test passes. GitHub CI on adbf649e16 completed with 60 successful checks, 15 skipped checks, and no failures or pending checks, including the vulnerability scan, all eight JS shards, all five Go test shards, macOS/Windows builds, WASM, and Go canary.

Approved upstream-only gate exception: make agent-report-progress passed Go/custom lint, JS lint (0 errors, 228 warnings), 5 impacted Go tests, 376 impacted JS tests, and compilation of all 365 workflows. It failed only on description drift in unchanged ponytail-reviewer.lock.yml, pr-code-quality-reviewer.lock.yml, and pr-specification-reviewer.lock.yml, introduced by main commit f33f7dccc7 (#76610). Those unrelated locks remain outside this PR.

env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" make fmt
npm --prefix actions/setup/js exec -- vitest run actions/setup/js/work_queue_replay.test.cjs actions/setup/js/work_queue_checkpoint.test.cjs actions/setup/js/work_queue_checkpoint_regression.test.cjs actions/setup/js/work_queue_claim_scope.test.cjs actions/setup/js/work_queue_delivery.test.cjs actions/setup/js/work_queue_delivery_proof.test.cjs actions/setup/js/work_queue_native_creation_delivery.test.cjs actions/setup/js/work_queue_specialized_delivery.test.cjs actions/setup/js/work_queue_rest_adapter.test.cjs actions/setup/js/work_queue_graphql_adapters.test.cjs actions/setup/js/work_queue_git_tree_adapters.test.cjs actions/setup/js/work_queue_git_adapter_parity.test.cjs actions/setup/js/work_queue_claim_adapters.test.cjs actions/setup/js/work_queue_control_receipts.test.cjs actions/setup/js/upload_artifact.test.cjs actions/setup/js/upload_artifact_work_queue.test.cjs actions/setup/js/create_code_scanning_alert.test.cjs actions/setup/js/upload_code_coverage.test.cjs actions/setup/js/safe_output_routing_parity.test.cjs actions/setup/js/finish_work_queue_claim.test.cjs actions/setup/js/work_queue_reconciler.test.cjs actions/setup/js/work_queue_delivery_evidence.test.cjs actions/setup/js/work_queue_delivery_evidence_restart.test.cjs actions/setup/js/work_queue_delivery_evidence_native_restart.test.cjs actions/setup/js/work_queue_delivery_evidence_adapters_restart.test.cjs --maxWorkers=2
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" go test ./pkg/workqueue -count=1
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" make agent-report-progress
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" make security-govulncheck
bash scripts/check-go-toolchain_test.sh
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" go test ./pkg/workflow -run '^(TestCompileWorkflowGoModVersion|TestGenerateRuntimeSetupSteps|TestGoRootCaptureReplacesStaleToolchainRoot|TestSerenaGoRuntimeExposure|TestAiderCoverageImprovementGates|TestAiderDebtDiscoveryOnlySelectsSourceMarkers)$' -count=1
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" make recompile
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" make agent-report-progress-no-test

Structure checks: exactly the manager and recovery imports; zero obsolete symbols; three pre-existing checkpoint test files, no split production files. Reproduce with:

grep -rn 'require("./work_queue_delivery_evidence.cjs")' actions/setup/js --include='*.cjs' --exclude='*.test.cjs' --exclude='*test_helpers.cjs'
grep -rln 'durableClaimReceipt\|verifyDurableOutput\|authorizeDurableClaimReadback' actions/setup/js pkg docs
ls pkg/workqueue/checkpoint_*.go

Publication checks (seven commits: six DeliveryEvidence behaviors plus the CI repair):

env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" gh pr view 76718 --json commits --jq '.commits | length'
env PATH="/tmp/gh-aw-go/root/usr/lib/go-1.26/bin:/tmp/gh-cli:$PATH" gh pr view 76718 --json body --jq .body | grep -c '^| B[1-6]'

ADR-76718

receipt = { expectation, expected_digest: digest(adapterEffectFields(adapter, stored.message)) };
}
if (!receipt || Buffer.byteLength(stored.adapter, "utf8") > 8192) return null;
return { verifier, adapter: stored.adapter, receipt };
Comment thread pkg/workqueue/request.go
"producer": {"pool_install", "deployment", "submit", "cancel_work"},
"dispatcher": {"pool_install", "deployment", "submit", "dispatch_next", "observe", "dispatch"},
"worker": {"submit", "dispatch_next", "observe", "finish", "dispatch"},
"worker": {"submit", "dispatch_next", "observe", "finish", "dispatch", "delivery_evidence"},
"minimum": 0,
"type": "integer"
},
"receipt": {
@github-actions

Copy link
Copy Markdown
Contributor

Category: bug | Risk: medium | Priority: medium | Score: 65/100 (impact 35 + urgency 17 + quality 13)
Recommended action: batch_review | Batch: pr-batch:runtime-safety

Generated by 🔧 PR Triage Agent · copilot · mai11 · ~6 AIC · ⌖ ~2 AIC · ⊞ 15.9K · ◷

@elefthei
elefthei force-pushed the fix/wq/durable-delivery-evidence branch from adbf649 to 5a1fe7a Compare November 11, 2026 21:07
@elefthei
elefthei marked this pull request as ready for review November 11, 2026 21:17
@elefthei
elefthei requested review from pelikhan and a dynamic review from Copilot November 11, 2026 21:17
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~3 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

Copy link
Copy Markdown
Contributor

✂️ Ponytail Reviewer has started processing this pull request

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

PR Specification Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Specification Reviewer · gpt56 · ~5 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

Generated by PR Specification Reviewer for #76718

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-11T21:24:00Z
review_event: COMMENT
top_themes: [durable receipt field mismatch causing unsupported restart verification]
files_reviewed: [actions/setup/js/work_queue_code_scanning.cjs, actions/setup/js/work_queue_delivery_evidence.cjs, actions/setup/js/work_queue_delivery.cjs]
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one blocking correctness issue in durable code-scanning evidence: the validator checks receipt.sha256 although receipts store finding_digest, so restart-safe evidence is never published for code-scanning outputs. The inline comment identifies the fix. Specialist pass was unavailable because its file-read tool call was denied; this review is based on the pre-fetched diff and existing comments.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review to run again

receiptMatchesClaim(receipt, claim) &&
["repository", "revision", "ref", "driver", "category", "sarif_id"].every(key => typeof receipt[key] === "string" && receipt[key].length > 0) &&
/^[A-Za-z0-9_-]+\/[A-Za-z0-9._-]+$/.test(receipt.repository) &&
/^[a-f0-9]{64}$/.test(receipt.finding_digest) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main() stores the receipt field as finding_digest (line 95), but this validator checks receipt.sha256, which is never populated. Consequently every code-scanning receipt fails validCodeScanningReceipt, durableCodeScanningReceipt() returns null, and collectDeliveryEvidence() publishes delivery_evidence_unsupported:create_code_scanning_alert; maintenance can never perform restart-safe verification for this output type. Validate finding_digest here (and retain the digest check below).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current branch already validates receipt.finding_digest, matching the stored field. The existing specialized-delivery test covers the 64-hex digest and durable verification, so no additional code change was needed.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate — ADR Verified

ADR reviewed: ADR-76718: Retain durable Claim delivery evidence — implementation aligns with the stated decision. Great work! 🏗️

📋 Verification Summary

The ADR Decision commitments are implemented as written:

  • One authenticated evidence publish after the handler pass: safe_output_handler_manager.cjs calls collectDeliveryEvidence and publishDeliveryEvidence in work_queue_delivery_evidence.cjs.
  • Bounds (protocol constants): 32 KiB canonical size, 128 outputs, 256 effects, and 8 KiB adapter fields, enforced in work_queue_delivery_evidence.cjs and pkg/workqueue/delivery_evidence.go.
  • Failed, unsupported, or oversized passes publish nothing: collectDeliveryEvidence returns unsupported, and the handler manager skips publication.
  • Immutability: identical republication is idempotent. Different evidence for the same Completion fails with delivery_conflict (work_queue_replay.cjs).
  • Maintenance: reconciler authorization with allowReleased: true in finish_work_queue_claim.cjs. One final readback is capped at 15000 ms via verifyWithinBudget. Recovery is assembled in work_queue_delivery_recovery.cjs.
  • Checkpoints: the delivery_evidence key is omitempty and written only when non-empty (pkg/workqueue/checkpoint.go). Result or DeliveryFailure removes pending evidence.
  • In-process path retained: the WeakMap receipt path is kept, and no diff lines remove WeakMap usage. Shared receipt selection and serialization (durableReceipt, claimDeliveryReceipt) live in work_queue_claim_scope.cjs.
  • Spec and schemas: specs/work-queue/transactions.tsp and generated pkg/workqueue/schema/DeliveryEvidence*.json. No TLA model files changed, which matches the ADR.

Divergences: none found.

Notes for maintainers (not divergences):

  • The ADR is still Draft with Deciders: Pending maintainer review. Per the ADR convention, a human should finalize it and update the status before merge.
  • The CI repair bumps the Go toolchain to 1.27.4 for GO-2026-6859. This is a dependency and toolchain change rather than an architectural decision, so it falls outside this ADR and does not need its own ADR.

Verification was limited to static inspection of the PR diff and the ADR. Tests were not run.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · routed: haiku55 xhigh · ~3 AIC · ⌖ ~2 AIC · ⊞ 8.5K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd and /grill-with-docs (selected by triage as new_feature). Posting observations, not blocking changes. The main open question is the controls_digest skip.

📋 Key Themes & Highlights

Key Themes

  • Restart-verification gap (/tdd): an omitted controls_digest skips the control-inventory check on recovery. Inline comment on work_queue_delivery_recovery.cjs:43.
  • Trust boundary not documented (/grill-with-docs): adapter config is persisted into the ledger. Low risk because it is closed and trusted, but the ADR should say so.
  • Silent publish degradation (/diagnosing-bugs): a failed evidence publish only logs a warning. The fallback to DeliveryFailure is safe, but a regression would be easy to miss.
  • Receipt schema is open (not inline): DeliveryEvidenceOperation.json uses additionalProperties: {}. Consider closed per-verifier receipt schemas. Verifiers re-validate receipts, so this is defense-in-depth, not an exploit.

Positive Highlights

  • ✅ Scope binding is enforced via scopedClaim (dispatch and run attempt), and the per-kind permission allowlist is correct.
  • ✅ Size bounds (32 KiB, 128 outputs, 256 effects, 8 KiB adapter) are enforced both in Go and in JS replay.
  • ✅ Negative-scope and immutability tests exist (delivery_evidence_test.go, work_queue_delivery_evidence.test.cjs).
  • ✅ go test ./pkg/workqueue/... passes.
  • ✅ The ADR captures the decision and its trade-offs.

Limits of This Review

  • The JS vitest suite was not run. npm ci failed on the sandbox TLS proxy, so the JS side was reviewed by reading only.
  • The base-commit diff was unavailable (partial clone), so the review used the final file state and the capped 3000-line patch.
  • The declared_builtin routing through verifyDeclaredOutput was checked by reading only.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~12 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

if (evidence.unsupported) core.info(`Claim ${handle}: durable delivery evidence not published (${evidence.unsupported})`);
else await publishDeliveryEvidence({ assignment: scope.assignment, claim_handle: handle, ...evidence, github: options.queueClient || github, context: verificationContext, core });
} catch (error) {
core.warning(`Claim ${handle}: durable delivery evidence publication failed: ${getErrorMessage(error)}`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] A failed evidence publish only logs a warning. Recovery then degrades to a DeliveryFailure, which is a safe fallback, but nothing makes a publish regression visible.

💡 Suggested change

Surface publish failures at a higher severity, or count them in run output, so a restart-safety regression shows up in CI. Add a test that asserts the warning path leaves the claim in DeliveryFailure rather than silently succeeding.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in d39c8e4: evidence-publication failures now emit an error annotation, and a restart test confirms no Result is published and the claim eventually settles as DeliveryFailure.

receipt = { expectation, expected_digest: digest(adapterEffectFields(adapter, stored.message)) };
}
if (!receipt || Buffer.byteLength(stored.adapter, "utf8") > 8192) return null;
return { verifier, adapter: stored.adapter, receipt };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] This persists the adapter config into the durable ledger. The risk is low because validateAdapter limits it to closed keys and a fixed target-repo, but that trust boundary is not stated.

💡 Suggested change

State in the ADR (docs/adr/76718-durable-claim-delivery-evidence.md) that adapter config is trusted, bounded, and persisted by design. Then any future adapter key that could carry a secret gets reviewed against that rule.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented in d39c8e4 that adapter objects are persisted only after closed-key validation and the 8 KiB bound; future fields that could carry credentials or personal data require review.

requireControlInventory: true,
readControlInventory: async () => {
const inventory = await readDeliveryControlInventory({ ...options, assignment, claim_handle: member.handle });
if (evidence.controls_digest !== undefined && inventory.controls_digest !== evidence.controls_digest) throw new Error("work_queue_control_inventory_changed");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] When controls_digest is absent, this check is skipped, so a restarted run can accept evidence without verifying the control inventory.

💡 Suggested fix

Require controls_digest whenever the evidence carries queue_control verification. If omission is intentional, document why it is safe and add a test that pins the accepted-without-digest behavior. The existing Go test at delivery_evidence_test.go:89-91 only covers the acceptance path.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in d39c8e4: queue-control evidence without controls_digest now fails closed before inventory readback, with restart regression coverage.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical recovery-verification gaps can promote rejected or failed deliveries to successful Results and incorrectly terminate valid deliveries as DeliveryFailures.

9 open issues
What changed in this PR

Adds durable delivery evidence to gh-aw’s work queue so maintenance can verify completed Claims after worker process exit without repeating writes.

Changes:

  • Defines bounded, worker-authenticated evidence with matching schemas, replay, and checkpoint support.
  • Adds receipt serialization, maintenance readback, and restart-focused verification coverage.
  • Documents the recovery contract and Go 1.27.4 security baseline.
File Description
specs/​work-queue/​transactions.tsp Defines evidence operations and receipt contracts.
scripts/​check-go-toolchain_test.sh Extends accepted Go patch-version coverage.
pkg/​workqueue/​types.go Adds evidence to projection state.
pkg/​workqueue/​selection.go Preserves evidence when cloning projections.
pkg/​workqueue/​schema/​RequestParameters.json Adds evidence request parameters.
pkg/​workqueue/​schema/​QueueRequest.json Allows evidence requests.
pkg/​workqueue/​schema/​QueueOperation.json Includes the evidence operation.
pkg/​workqueue/​schema/​QueueCommit.json Includes evidence in commit validation.
pkg/​workqueue/​schema/​OperationsParameters.json Includes evidence operation parameters.
pkg/​workqueue/​schema/​DeliveryEvidenceWriteOutput.json Defines write-receipt validation.
pkg/​workqueue/​schema/​DeliveryEvidenceOutput.json Defines the output receipt union.
pkg/​workqueue/​schema/​DeliveryEvidenceOperation.json Defines evidence identity and bounds.
pkg/​workqueue/​schema/​DeliveryEvidenceNoWriteOutput.json Defines no-write output entries.
pkg/​workqueue/​schema/​DeliveryEvidenceExpected.json Defines effect readback expectations.
pkg/​workqueue/​schema/​DeliveryEvidenceEffect.json Defines recorded effect metadata.
pkg/​workqueue/​request.go Permits worker evidence requests.
pkg/​workqueue/​replay.go Applies evidence and clears terminal entries.
pkg/​workqueue/​policy.go Initializes evidence and enforces worker-only publication.
pkg/​workqueue/​policy_permitted_test.go Tests evidence operation permissions.
pkg/​workqueue/​delivery_evidence.go Validates evidence scope, immutability, and bounds.
pkg/​workqueue/​delivery_evidence_parity_test.go Tests cross-runtime replay and restart parity.
pkg/​workqueue/​checkpoint.go Retains and validates pending evidence.
docs/​src/​content/​docs/​specs/​work-queue-specification.md Specifies durable verification and expiry behavior.
docs/​src/​content/​docs/​reference/​work-queue.md Documents maintenance recovery behavior.
docs/​adr/​76718-durable-claim-delivery-evidence.md Records design decisions and tradeoffs.
CHANGELOG.md Documents the patched Go baseline.
actions/​setup/​js/​work_queue_upload_assets.cjs Adds serializable asset receipts and readback.
actions/​setup/​js/​work_queue_specialized_delivery.test.cjs Tests specialized receipt serialization and readback.
actions/​setup/​js/​work_queue_rest_adapter.test.cjs Tests durable REST receipts and bounds.
actions/​setup/​js/​work_queue_rest_adapter.cjs Adds receipt-explicit REST verification.
actions/​setup/​js/​work_queue_replay.test.cjs Tests evidence replay and checkpoint lifecycle.
actions/​setup/​js/​work_queue_replay.cjs Adds JavaScript evidence validation and retention.
actions/​setup/​js/​work_queue_reconciler.cjs Supports assignment-specific verifier factories.
actions/​setup/​js/​work_queue_policy.cjs Allows worker evidence publication.
actions/​setup/​js/​work_queue_native_creation_delivery.test.cjs Tests serialized native-creation receipts.
actions/​setup/​js/​work_queue_native_creation_delivery.cjs Adds digest-based discussion and PR receipts.
actions/​setup/​js/​work_queue_graphql_adapter.cjs Adds durable GraphQL receipt verification.
actions/​setup/​js/​work_queue_graphql_adapter_checks.cjs Tests GraphQL restart readback and bounds.
actions/​setup/​js/​work_queue_git_tree_adapter.cjs Adds digest-based durable Git receipts.
actions/​setup/​js/​work_queue_git_tree_adapter_checks.cjs Tests Git receipt recovery and mutation rejection.
actions/​setup/​js/​work_queue_delivery_recovery.cjs Assembles read-only maintenance verification.
actions/​setup/​js/​work_queue_delivery_evidence.test.cjs Tests evidence collection and publication.
actions/​setup/​js/​work_queue_delivery_evidence.test_helpers.cjs Provides worker/restart maintenance fixtures.
actions/​setup/​js/​work_queue_delivery_evidence.cjs Collects, publishes, and routes durable receipts.
actions/​setup/​js/​work_queue_delivery_evidence_restart.test.cjs Tests restart settlement without repeated writes.
actions/​setup/​js/​work_queue_delivery_evidence_native_restart.test.cjs Tests discussion and PR recovery after restart.
actions/​setup/​js/​work_queue_control_receipts.test.cjs Tests trusted control-inventory verification.
actions/​setup/​js/​work_queue_control_receipts.cjs Adds committed queue-control readback.
actions/​setup/​js/​work_queue_code_scanning.cjs Adds digest-based SARIF delivery receipts.
actions/​setup/​js/​work_queue_code_coverage.cjs Adds durable coverage receipts and validation.
actions/​setup/​js/​work_queue_claim_scope.test.cjs Tests explicit durable receipt rebinding.
actions/​setup/​js/​work_queue_claim_scope.cjs Shares receipt selection and serialization helpers.
actions/​setup/​js/​work_queue_claim_adapters.test.cjs Tests durable declared-builtin adapters.
actions/​setup/​js/​work_queue_claim_adapters.cjs Adds durable adapter export and verifier construction.
actions/​setup/​js/​upload_artifact.cjs Adds artifact receipt serialization and readback.
actions/​setup/​js/​upload_artifact_work_queue.test.cjs Tests artifact receipts after module reload.
actions/​setup/​js/​safe_output_handler_manager.cjs Publishes evidence after worker handler passes.
actions/​setup/​js/​reconcile_work_queue_run.cjs Enables maintenance recovery and reporting.
actions/​setup/​js/​finish_work_queue_claim.cjs Adds reconciler authorization and final readback handling.
.changeset/​compiler-transport-trust.md Updates the documented Go baseline.

Review effort: Max (auto)

configured.core?.warning("Work queue obsolete-work reconciliation failed; core recovery will continue. Inspect safe API diagnostics and the checked ledger before retrying maintenance.");
}
const result = await reconcileQueue({ ...configured, requestCancellation: false, expireUnverified: true });
const result = await reconcileQueue({ ...configured, requestCancellation: false, expireUnverified: true, verifyEffectsFor: options.verifyEffectsFor || (assignment => createDurableDeliveryVerifier({ ...configured, assignment })) });
configured.core?.warning("Work queue obsolete-work reconciliation failed; core recovery will continue. Inspect safe API diagnostics and the checked ledger before retrying maintenance.");
}
const result = await reconcileQueue({ ...configured, requestCancellation: false, expireUnverified: true });
const result = await reconcileQueue({ ...configured, requestCancellation: false, expireUnverified: true, verifyEffectsFor: options.verifyEffectsFor || (assignment => createDurableDeliveryVerifier({ ...configured, assignment })) });
Comment on lines +2059 to +2063
if (options.registerVerification !== undefined && !isStagedMode()) {
try {
const evidence = collectDeliveryEvidence({ assignment: scope.assignment, claim_handle: handle, messages, results, effects: options.effects || [], controls, config: loadConfig(), context: verificationContext });
if (evidence.unsupported) core.info(`Claim ${handle}: durable delivery evidence not published (${evidence.unsupported})`);
else await publishDeliveryEvidence({ assignment: scope.assignment, claim_handle: handle, ...evidence, github: options.queueClient || github, context: verificationContext, core });
Comment on lines +849 to +850
async function verifyArtifactReceipt({ claim, github }, receipt) {
if (!validArtifactReceipt(receipt, claim)) return { verified: false };
Comment on lines +34 to +39
if (NO_WRITE_TYPES.has(type)) return { entry: { index, type, verifier: "none" } };
const outcome = results.find(result => result.messageIndex === index);
// Omitting a failed optional output would turn an incomplete pass into a
// complete synthetic inventory after restart. Never publish that subset.
if (!outcome || !outcome.success || outcome.delegated || outcome.deferred || outcome.skipped || outcome.cancelled) return { unsupported: `delivery_evidence_receipt_missing:${type}` };
if (outcome.claim_handle && outcome.claim_handle !== claim.handle) throw new Error("work_queue_claim_scope_invalid");
if (message.claim_handle !== claim.handle || !["work_queue_submit", "work_queue_dispatch_next"].includes(message.type)) throw new Error("work_queue_control_receipt_scope_invalid");
// request_id is filled from authenticated worker authority at publication,
// not the unauthenticated Actions context supplied to the handler manager.
outputs.push({ index: messages.length + offset, type: message.type, verifier: "queue_control", receipt: { intent_id: message.intent_id } });
return inventory;
},
verifyDeclaredOutput: createDurableDeclaredVerifier(evidence.outputs, effects),
verifyOutput: verifyEvidenceEntry,
],
Claim: [["kind", "work_id", "claim_id", "dispatch_id", "handle", "observations"], []],
Completion: [["kind", "work_id", "claim_id", "dispatch_id", "claim_handle", "run_id", "run_attempt"], ["keep_open"]],
DeliveryEvidence: [["kind", "work_id", "claim_id", "dispatch_id", "claim_handle", "completion_id", "run_id", "run_attempt", "outputs", "effects"], ["controls_digest"]],
reconciled: result.reconciled,
released,
unresolved,
durable_verified: result.results.filter(entry => Object.values(entry.delivery?.claims || {}).some(claim => claim.state === "result")).length,
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/work_queue_code_scanning.cjs:110): address unresolved feedback - Review comment
  3. Review (actions/setup/js/safe_output_handler_manager.cjs:2065): address unresolved feedback - Review comment
  4. Review (actions/setup/js/work_queue_claim_adapters.cjs:287): address unresolved feedback - Review comment
  5. Review (actions/setup/js/work_queue_delivery_recovery.cjs:43): address unresolved feedback - Review comment
  6. Fix failing check conclusion (FAILURE): Check run.

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "5a1fe7a7b7f854b1fffd80150d215addbe1b1b70",
  "kind": "requested",
  "schema_version": "1",
  "state": "7a7d724d577d90e1f70efd22533ab8540c0b7833bbc5c62a431c455529508934",
  "work": [
    "14d1bff15b137e43e6d04c254a5005af5064357e69aa4d379c56d9ebd9780c3a",
    "51e2287c3d1ee41b7e13769862add61dcd292a623cf943abc5968c984e246bcf",
    "ae9e6da09f42352b7fb62950a192aa6069720befae086ea83c724bcafabad2ab",
    "ee1e3f66cbe756e4582e3ef4d93cf09cfe41bc8042de8dfe99033b074949ee7c",
    "f2a1d74d275a90c1cd44b26a5be86d15ba58b0cfefb0980afac2aa391410b7d4"
  ],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 high · ~2 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

@elefthei
elefthei force-pushed the fix/wq/durable-delivery-evidence branch from d39c8e4 to 9e1bb02 Compare November 11, 2026 23:03
@elefthei
elefthei merged commit ed04d06 into main Nov 11, 2026
67 checks passed
@elefthei
elefthei deleted the fix/wq/durable-delivery-evidence branch November 11, 2026 23:19
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants