Skip to content

feat(runtime): establish Phase 3A recovery persistence authority - #1521

Merged
likun666661 merged 19 commits into
apache:mainfrom
zhiiw:codex/runtime-recovery-authority-v2
Jul 27, 2026
Merged

likun666661 merged 19 commits into
apache:mainfrom
zhiiw:codex/runtime-recovery-authority-v2

Conversation

@zhiiw

@zhiiw zhiiw commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This is the first flat replacement slice extracted from the #1346 integration experiment. It establishes the Phase 3A recovery persistence authority without bringing file reconciliation, continuation execution, or host wiring into the same review boundary.

The invariant proved by this PR is:

Recovery facts have one atomic persistence authority, and online projection, close/reopen, projection rebuild, and RecoveryResolver derive the same terminal interpretation from immutable RuntimeEvents.

What changed

  • Added a lossless canonical RuntimeEvent codec and strict-JSON tool argument validation while preserving the deployed t1_after_preflight_v1 hash bytes. Any future domain-separated hash requires an explicit versioned dispatch protocol.
  • Defined exact call, dispatch, outcome, reconcile, and decision semantic lanes with one shared scanner and prospective transition validator.
  • Added exact v1 reconcile-result and recovery-decision facts plus a shared causal bundle interpreter.
  • Made one SQLite transaction the only authority for reconcile + optional synthesized outcome + terminal decision.
  • Made completed decisions require an identity-matching persisted outcome; parked is a permanent v1 terminal state with exact retry only.
  • Made SQLite tool projections disposable and rebuildable solely from immutable RuntimeEvents.
  • Made writer, rebuild, Resolver, and resume gates consume the same scanner/interpreter instead of parallel state machines.
  • Hardened JSONL fallback writes with reserved-fact gates, target Run identity validation, physical exact-retry deduplication, stable-storage retry, steering-proof convergence, and partial cleanup.
  • Hardened SQLite multi-process startup and tests: bounded worker coordination, immediate exit capture, concurrent WAL opener coverage, and no repeated setting-form journal_mode=WAL on established databases.

Safety model

  • Immutable RuntimeEvents are the semantic authority; SQLite journal/projection rows are derived state.
  • Generic append/import paths cannot smuggle reserved durable tool or recovery facts.
  • Immutable corruption fails closed. Mutable partial corruption remains fail-soft and cannot alter durable high-water.
  • One invocation may map to only one (sessionId, runId, turnId) execution spine.
  • The first implementation intentionally uses a workspace-wide semantic fail-stop: corruption in any canonical tool ledger blocks later tool-bearing boundaries in that SQLite workspace. This favors correctness over fault-domain size and write-path cost; a rebuildable incremental reducer is the planned optimization.
  • Mainline schema 4 upgrades to schema 5. Existing t1_after_preflight_v1 dispatches retain the exact deployed mainline hash bytes, including the historical required/enum ordering and __proto__ behavior. Unreleased feat(runtime): complete Phase 3A file-transaction recovery #1346 experimental recovery data is explicitly unsupported rather than guessed or migrated.

Deliberate non-goals

This PR does not add:

  • recovery contract registries, observers, or reconcilers;
  • Write/Edit file checkpoints or automatic redo;
  • continuation claims, provider replay, or automatic resume;
  • Desktop/CLI resume wiring or host lifecycle changes;
  • Git checkpoint carriers, restricted verification, retry, or reattach prototypes.

Those remain independent follow-up slices so each PR can prove and roll back one invariant.

Validation

After rebasing onto upstream/main@466f238b:

  • npm run build — passed across all workspaces, including the Desktop renderer build.
  • npm run typecheck — passed across all workspaces.
  • Recovery authority, schema-4 migration, deterministic stale-read, multi-process opener, Resolver, equivalence, commit-sink, and real SQLite boundary suites — 79/79 passed.
  • Frozen mainline-v1 literal vectors cover ordinary arguments, required, nested enum, ordinary arrays, and historical __proto__ behavior.
  • A real schema-4 call + t1_after_preflight_v1 dispatch fixture rebuilds successfully under schema 5.
  • The full Storage suite terminates instead of hanging. The Windows local run still exposes existing unrelated platform failures around symlink privileges, SQLite file cleanup EBUSY, and source-path fixtures; none are in the PR A focused suites or changed paths.
  • POSIX SIGKILL durability remains a Linux/macOS CI proof requirement; Windows support remains intentionally limited for those crash cases.

Rebase note

The branch is rebased onto upstream/main@466f238b. The only content conflict was an incidental overlap in the Desktop settings E2E fixture; the resolution retained upstream's newer exact three-action permission contract, so that file contributes no PR A diff. git range-diff keeps the remaining recovery-authority commits and both schema-4 blocker fixes semantically equivalent.

Relationship to #1346

#1346 remains the integration/design record. This PR replaces only its recovery-persistence-authority slice. #1346 should remain Draft until the continuation and conservative prepared-file replacement slices are merged as separate PRs.

Detailed extraction and ownership ledger: docs/architecture/runtime-resume-extraction-ledger.zh-CN.md.

@likun666661

Copy link
Copy Markdown
Member

Review findings (blocking):

  1. Schema 4 T1 events become corrupt after the tool-args hash change.

    Main currently persists canonicalArgsHash as stableHash({ toolName, args }), while this PR adds the canonical_tool_args_v1 domain to the hash input. Migration 5 only adds the capability marker; it does not version or migrate existing dispatch facts. After upgrading a schema 4 database containing a real T1 dispatch, scanToolLedger recomputes the new hash and rejects the previously valid event.

    I reproduced this with a schema 4 call + dispatch written using the mainline hash. After opening it with this branch, projection rebuild fails with:

    Corrupt tool RuntimeEvent ledger: canonical_args_hash_conflict at dispatch-event-1
    

    Because tool-bearing writes scan the workspace ledger, the same historical row also fail-stops later tool boundaries. The current schema-upgrade test only covers legacy projections without a dispatch RuntimeEvent, so it misses the deployed schema 4 shape.

    Please either preserve the existing hash semantics for t1_after_preflight_v1, or introduce an explicit dispatch protocol/hash epoch and teach the scanner to validate both versions. A migration test containing an actual schema 4 dispatch fact would catch this.

  2. Concurrent schema 4 → 5 migration can make one opener fail.

    migrateSqliteRuntimeDatabase reads user_version before acquiring BEGIN IMMEDIATE. Two processes can both cache version 4, then serialize on the lock; after the first creates runtime_capabilities, the second still executes migration 5 and fails.

    Controlled reproduction: hold a write lock, start two openers against the same schema 4 database so both read version 4, then release the lock. One succeeds and the other exits with:

    Error: table runtime_capabilities already exists
    

    The new concurrent-opener test uses an already initialized schema 5 database, so it does not exercise this path. Please acquire the migration lock first and re-read user_version under that lock (or otherwise make migration application transactional and re-checkable), and add a concurrent-upgrade test.

The focused 75-test recovery/storage/runtime suite and the three current CI checks pass; neither scenario above is covered by them.

@zhiiw
zhiiw force-pushed the codex/runtime-recovery-authority-v2 branch from a188a2d to 1402c6b Compare July 27, 2026 14:07
@zhiiw

zhiiw commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both blockers in 1402c6b6 (rebased head; the pre-rebase fix was a188a2d9).

  1. Schema-4 T1 identity compatibility

    • t1_after_preflight_v1 now validates arguments as strict JSON while computing identity with a frozen reproduction of the deployed mainline stableHash({ toolName, args }) bytes.
    • This preserves object-key ordering, legacy required/enum array sorting, ordinary-array order, and the historical __proto__ omission. Any domain-separated semantics are reserved for an explicit future dispatch/hash protocol version.
    • Literal compatibility vectors cover ordinary arguments, required, nested enum, ordinary arrays, and __proto__.
    • The migration test now contains a real schema-4 function call and T1 dispatch using the special required vector, then verifies that schema 5 rebuilds the prepared projection successfully.
  2. Concurrent schema migration

    • A pending upgrade now takes BEGIN IMMEDIATE, re-reads user_version while holding that lock, and applies all remaining migrations in the same transaction.
    • A deterministic stale-read regression simulates an optimistic v4 read followed by a locked v5 read and verifies that migration-5 SQL is not executed again.
    • The real two-process schema-4-to-5 opener test remains as an integration smoke test.

The branch has also been rebased onto upstream/main@466f238b. The PR is mergeable, and the rebased head passed all three CI checks (typecheck, test, and e2e). Local root build/typecheck pass as well, with the focused recovery/schema/concurrency/runtime suite at 79/79.

@zhiiw
zhiiw marked this pull request as ready for review July 27, 2026 15:12
@likun666661
likun666661 merged commit 086ec99 into apache:main Jul 27, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants