Skip to content

docs(operations): prepare Session cutover and release evidence #640

Description

@Yevanchen

Physical observation evidence — September 22, 2026

Source eb4ff3ab9d in PR642 adds a non-starting internal physical observation path. It resolves the recorded Sandbox namespace and ID, then reads platform running state without SDK initialization, keep-alive configuration or DO storage access. Unavailable state and transport failures cannot masquerade as stopped. Normal access still restores network policy; first-time teardown now consumes its expected restoration rejection without enabling access.

Validation: local API 1,179 pass /4,601 assertions, six isolated wrapper cases and all four namespace routes; typecheck, lint, format, 62 document links and public API checks pass. A synthetic local container on actual SDK0.12.6/Driver9ffe passes cold/warm/stopped observation, real file write/read and exec, unchanged observation storage/alarms, and final container/Worker cleanup. Earlier harness failures remain recorded. Exact-source full Linux CI passes; local full validation retains unchanged macOS Driver failures (1,253 pass /61 skip /11 fail /1 error).

This supplies physical observation, not an exclusion/drain barrier or completed production executor. A later request or existing SDK Session closure can still access the resource. The 28 formal workspace and 6 unexpired Preview configuration source gaps remain; no Cloud conversion or recovery is claimed. The existing staging rewrite/deploy approval packet stays fixed at70bf and still awaits its answer. No staging/production deployment, Cloud data write, model call or customer execution occurred; only the synthetic local fixture was created and destroyed. #582 and all scoped children remain OPEN; #636 remains excluded.

Cutover guard validation — September 22, 2026

Source dffc47142f in PR642 closes a reproduced migration guard gap: active Runs can still reference a Sandbox through workspace membership or a stopped/failed Driver even when Agent provenance is absent. The forward batch and both rollback resources now check those actual relationships atomically. Unrelated direct Runs remain unaffected; normal Run termination releases the guard. No customer Run is cancelled to force eligibility.

Validation: 42 planner tests /113 assertions; local API 1,176 pass /4,580 assertions; API/workspace typechecks, lint, format, 62 document links and public OpenAPI/v1 compatibility pass. Local workerd replays all 18 migrations and verifies nine stale/busy rejections without row changes, injected mid-batch atomic failure, and functional forward/rollback reader selection. Exact-source full Linux CI passes. Local full check retains macOS Driver failures (1,252 pass /61 skip /12 fail /1 error); the Driver pin is unchanged.

The previous new-admission/UI type-retirement source 8bfb7f also passed its full Linux CI. Every new Session is isolated; historical shared bindings and legacy maintenance paths remain until verified Cloud conversion. This is still not physical container-drain proof or a complete production executor. The 28 formal workspace and 6 unexpired Preview configuration source gaps remain unresolved. Staging's concrete 70bf rewrite/deploy packet still awaits its existing approval; no substitution, remote deployment/data rewrite, model call or production operation occurred in this slice. #582 and all in-scope children remain OPEN; #636 remains excluded.

Current target and source — September 22, 2026

The approved primary path is Project key + explicit harness/provider/model + instructions/input/files → durable Session, with no pre-created Agent. A saved private Agent is an optional explicit preset; inline and preset configuration cannot be mixed. BYOK remains required; #636 platform model supply, recharge and commercial billing remain excluded. Earlier saved-Agent and staging observations below are source-specific history, not completion of the direct target.

Source 70bfda2a10 implements Project v2 creation/upload through the existing kernel, immutable configuration, nullable Agent provenance, same-request recovery after preset deletion, Project-scoped receipts across CLI login/key replacement, and attachment validation before prewarm. The generated contract, typed client and executable HTTP workflow are synchronized. Local API 1,160 tests, client 11 tests, workspace typecheck/lint, v1 compatibility and 61 document links pass; focused direct HTTP tests pass 23 cases. The exact-source Linux full repository gate passes.

This source is not deployed. Hosted staging evidence below remains c5b79cb / Driver47d63d26 (protocol 4). The candidate pins Driver9ffe8b16 with protocol 5 and appended nullable-provenance migration 0017. Matched Host/Driver rollout, direct real-tool/cold-continuation acceptance, external CLI/docs updates, Cloud migration, active Type retirement and approved release remain required. Existing formal/API-used Sessions retain the same-ID continuity guarantee. Cloud debug Preview has only the approved 30-day inactivity exception; historical cleanup still needs a concrete inventory, recoverable backup and production approval.

Driver provisioning migration fence — September 22, 2026

Source c7e8528a6a5ffa662cb7cd3803fa453ce40bf6f9, unchanged Driver 47d63d26c69d64e255cc2d1ee8d88fc8057ddbc5: exact-source Linux full repository gate passes. API 1,128 pass /4,313 assertions; Driver 1,283 pass /38 optional skips /3,771 assertions. Local API/typecheck, whole-repository lint/format, 60 document links and diff checks pass.

A delayed startup/prewarm previously could create or replace a Driver for an obsolete Session binding and access its workspace before the claim completed. Reproduced against the original implementation, then fixed: binding validation, Driver claim, command cleanup and MCP grant publication share one atomic D1 batch; workspace setup waits for the successful claim. Skipped claims cannot append grants, including same-token retries; a later grant failure preserves all original state. Actual local workerd D1 with all 17 migrations verifies these paths. Two real provisioning seam cases run in a separate test process to avoid the existing suite's global mock contamination; the original failed run is retained.

The existing conversion tests already exercise actual new-message admission in both orders: admitted work wins and rejects migration, or migration wins and admission uses the new committed binding. This patch is a necessary fence, not complete physical Driver-drain or production-executor proof. No schema, migration, generated/public contract, Driver pin, dependency or lockfile changes.

This new patch is not deployed. Hosted staging evidence below remains tied to c5b79cb; production data, deployments and customer execution are unchanged by this work. #582 and its six in-scope subissues remain open; #636 remains excluded.

Staging conversion verification — September 22, 2026

Mosoo c5b79cb3b00c4c6a88ea3116132189668b44eb01 and Driver 47d63d26c69d64e255cc2d1ee8d88fc8057ddbc5 are now deployed to isolated staging. API e89cc3d2-c2aa-4bb6-9815-4577c6c4a7e8 and Web 18fac9b1-b055-41a4-91b0-96b2c35db902 have matching source tags at 100%; all four stage image digests match. Both hosts pass deep health and exact generated v1/v2 OpenAPI checks. The exact-source Linux repository gate remains green: API 1,120 pass; Driver 1,283 pass /38 optional skips. No tracked source changed during this staging exercise.

One newly created synthetic Pet has now actually migrated and continued through the same public Session ID. Its cold workspace and corresponding shared-memory archive were inspected; canonical preparation and cold restoration preserve 71 durable files, modes, links, native state and cwd. The actual shared-memory source is empty. Full-schema local workerd tests exercise all 17 migrations, actual readers, stale/duplicate rejection, mid-batch atomic failure and functional rollback; the operator CLI generates identical batches. Two prepared/rollback archives and their metadata were uploaded under fresh IDs and byte-verified, followed by one guarded atomic remote D1 batch. Original resources remain retained.

The converted Session then completed a real tool call with gpt-5.6-luna at an estimated $0.003582, under a $0.05 turn threshold. The independently downloaded new committed checkpoint retains the entire original native rollout as an exact byte prefix, the same native ID, original random-file bytes, and precisely the requested five appended source bytes. Frozen admitted configuration, cwd, delegated origin and public artifact IDs remain consistent. Original v1 routes still read that ID, both turns and exact original artifact bytes. The old rollback guard rejects newly admitted work, preventing rollback from discarding the continuation.

Seven pre-existing synthetic Cattle fixtures also had external memory links repaired from their finite audited test histories before deployment: 533 durable files and 14 prepared/rollback archives were verified; only seven new maintenance backup references were inserted. One protected Luna cold continuation passed at $0.004222. That synthetic-only empty-memory qualification must not be generalized to customer sources. The new Pet baseline cost $0.003605 and its exact requested text/newline assertion failed; the failure is retained, and migration fidelity uses the actual independently verified source bytes. Total estimated model cost for this staging milestone: $0.011409; the old Claude $0.25 authorization was not reused. Network failures were reconciled against actual object contents; no uncertain D1 batch was replayed.

This is one synthetic hosted conversion, not a Cloud customer migration, complete physical admission/drain barrier, all-runtime live conversion, or production-ready executor. Existing six real Cloud copies retain their separate offline 17-migration/native/file/rollback evidence (7,769 durable files; 14 unaffected peer bindings); no customer model/tool call or production mutation occurred. The live canary uses the retained union Sandbox binding. Source-specific observations further below remain historical; this section supersedes earlier staging-version statements.

All seven scoped issues remain OPEN. Remaining work: production-grade admission/Driver drain and guarded execution; unresolved 30 Cloud workspace and 27 original extra-configuration paths; affected existing Cloud memory-link qualification; full active Pet/Cattle Type retirement; integrated CLI/docs release; and owner-approved production cutover. #636 commercial supply/recharge/billing remains excluded. Production default/maximum budget and the concrete cutover packet remain later owner decisions; no new decision is required to continue current engineering.

Current release scope — September 22, 2026

Configure Project-owned model credentials (BYOK), then invoke a durable Session directly with explicit harness/provider/model/instructions/input/files. Creating or publishing an Agent is not required. An owned saved Agent remains an optional preset; CLI/docs must lead with the direct path and preserve compatible v1 entry points. Keep usage records and per-turn budget guards. Platform model supply, recharge and commercial billing remain separate #636 work.

The latest read-only Preview classification observed September 22 identifies 58 inactivity candidates, 6 API/key-protected Sessions, and no active Runs/uploads at observation. It does not authorize deletion. After the approved Preview exception is applied to scope, 28 formal workspace gaps and 6 unexpired Preview configuration gaps still need treatment. No additional historical backups are known. Review the concrete candidate list, recoverable copies, cutover order and rollback before requesting production approval; existing Cloud continuity cannot be replaced by a new Session or an error. Production budget settings remain a release decision.

Parent: #582

Status: owner-approved delivery slice. Human cost/legacy/release decisions remain explicit gates where listed.

Blocked by

#638 and #639; production review/authorization under repository rules.

Acceptance criteria

Verified starting point — September 18, 2026

The completed #581 authentication cutover and notices do not cover this release. No #582 deployment or notification has occurred in this worktree. Full local check encountered 11 unchanged Driver process/watchdog test failures on macOS; resolve or verify the relevant Linux gate before release.

Owner clarification — September 18, 2026

Do not require an old-endpoint sunset or client upgrade solely for Thread/Run naming. Tie migration instructions and notices to measured Cloud customer behavior/data impact. Record source/Driver/Worker versions for matched API/Web staging, real tool/artifact/continuation evidence, and the specific production transition; never treat a staging deployment as production acceptance.

Technical acceptance update — September 18, 2026

The owner deferred production launch allowances/default model supply while continuing technical acceptance. API PR 642 at e060f465ac24a11fe7efe0cf55fde78042f42e2f passes the exact-source Linux repository gate and is deployed only to isolated staging. API/Web versions, Driver/container images, D1 migration 0016 and live OpenAPI were verified; production remains unchanged. Claude budget acceptance passed 21 assertions and new v2 Session isolation passed 23 assertions on the prior source with unchanged isolation code.

CLI PR 88 at c3d0610cc9fac7ca722029e46075996506da1beb and Docs PR 43 at c784584dea986eb829029b1e0b75a2165804c475 pin the immutable e060 contract and pass CI. The frozen CLI passed all nine general staging smoke checks, preserved budget rejection details, and completed a fresh API-to-CLI same-Session continuation with independently verified original file bytes. The three-language docs passed 28 runtime-reader examples, 48 tests, 169 built article/Markdown pairs and the full build. No CLI release/global installation or hosted docs production deployment occurred.

One earlier CLI wait exited nonzero despite eventual server completion; its diagnostics were not captured, so a later fresh pass is not a proven root-cause fix. The September 18 no-credits OpenAI failure is historical; see the recharged low-cost acceptance below. Actual multi-day live recovery, full Pet/Cattle transition, restore evidence for Cloud cohorts, production approval/release and required notices remain open. Direct managed first-use/default supply is separate #636 scope under the September 19 decision. No issue is closed by this acceptance update.

Current source and delivery — September 21, 2026

The active #582 goal tracks #634/#635/#637/#638/#639/#640. BYOK remains the approved launch scope; #636 platform model supply, recharge and commercial billing are independent and are not closure blockers.

  • API: PR642 0dd2c2b5cb1cf6feeb88ceb6cc6f0a34e811d804, Driver 47d63d26c69d64e255cc2d1ee8d88fc8057ddbc5. Native/callback protection and offline atomic conversion/rollback planning pass Linux full CI; merged local API 1,113 tests /4,270 assertions. Main fix(runtime): preserve checkpoints and retry accepted receipts #644 is integrated; this task has not deployed the feat(api): add the durable Agent Session API #582 candidate. Remote cohort qualification, legacy-memory conversion, coordinated protocol cutover and release remain required.

  • CLI: PR88, b51cea90e2ae2c4c7cc753ff1114141ac3e403f7, remains open/ready against main with generated-contract CI passing. No public release or global installation occurred.

  • Docs: PR43, feb32c74c74614f1ba421e01b1c86f50b0e127ea, remains open/ready against main with documentation/OpenAPI CI passing. No hosted production publication occurred.

  • Driver: PR126, 727932ea147740b3c9c0a3021a83fb15322b7988, is still open on its existing image-integration base. Complete its compatible main/release integration before the pinned release; do not treat an unchanged submodule pin as an integrated release.

  • Acceptance: actual Luna 49.0242-hour and Claude BYOK 61.6578-hour continuation passed; the old Claude CSV passed one separately approved cold retry at an estimated $0.177245. Prior failures remain failures and the one retry authorization is consumed. The public API-to-CLI continuation and exact file downloads retain their original evidence. The known macOS Driver full-check failures remain disclosed despite passing Linux gates.

  • Cloud copies and an important correction: two unarchived OpenAI copies passed offline native/file verification and in-memory D1 binding/rollback. The first workspace rehearsal used the wrong attachment exclusion path. A fresh rehearsal used the actual prepareRuntimeSessionWorkspaceCheckpoint implementation: both copies preserve 74 non-ephemeral files, not the previously reported 74/75. One copy excludes temporary auth; the other excludes auth plus one current-message attachment (three session-files entries). Canonical cold restore, synthetic memory persistence and original-archive rollback pass. The previous script and observation remain preserved as superseded evidence. Four more unarchived Claude/ACP copies passed native/file and canonical workspace restore/selected-source rollback; one used an independently verified older source because its latest archive was empty. None is a live customer migration or production functional rollback.

  • Latest ready does not imply recoverable: actual extraction of all 43 four-kilobyte latest archives in the 53-Session metadata-qualified cohort found empty roots. A fresh SELECT reconfirmed all 43 were unarchived, had an existing Agent, a completed latest Run/matching observed native reference and no active Run. All 33 retained older archives were then inspected: 18 were empty and 15 nonempty sources were created after the respective latest successful Runs. Fourteen match one native context; the remaining OpenAI source contains the expected parent plus three native child contexts with matching parent IDs, cwd and database/rollout identities. All 32 canonical user/assistant message texts are present in role order across those 15 sources. This does not yet prove every tool effect, shared file, configuration source or live recovery. The other 28 Sessions have no nonempty source among the retained workspace archives inspected; that is not proof of permanent data loss. No production backup selection or binding changed.

Remaining-source investigation: bounded read-only checks of the 28 unresolved empty-source Sessions plus the two separately identified successful Claude Sessions found 189 pruned workspace backup references. A fresh complete 820-object R2 listing found neither metadata nor archive objects for those references. The 30 Sessions have 30 Runs, 60 messages and 2,976 events: 8 have tool events and 9 have file records, with overlap; 19 have neither in these records (17 Claude, one ACP, one OpenAI). Canonical source rows for those 19 were privately preserved, with qualification rechecked. This classification does not prove absence of unrecorded workspace state or complete reconstruction. Existing bounded-history replay is not automatically accepted as full-context recovery. No customer model/tool ran and production writes remained zero.

The migration executor, admission/drain concurrency gate and functional production rollback remain unfinished. Exact row rollback is insufficient where the old reader would select a known-empty archive. Prepare concrete customer cohorts, verified source/rollback archives, compatible release ordering and customer notice before the final production approval. Production default/max turn-budget values also remain a launch decision; the staging values are not an approved production policy.

No production deployment, data rewrite, customer tool/model execution, notification, CLI release, hosted-docs release or issue closure occurred. Preserve old routes/IDs where behavior is compatible; naming alone creates no client-upgrade requirement. #640 and #582 remain open.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions