Skip to content

refactor(storage): replace replay Artifact paths with a SQLite context-offload store #4071

Closed
Task
Closed
refactor(storage): replace replay Artifact paths with a SQLite context-offload store#4071
Task
Assignees
likun666661

Description

@likun666661
likun666661
opened on Aug 28, 2026

Problem

Discussion #4030 separated three decisions that the current ArtifactStore conflates:

  1. the immediate causal fix for fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027;
  2. logical ownership and product surfaces;
  3. the physical byte-storage substrate.

This issue owns one cohesive domain only: Agent context offload. A model-visible value is too large to keep inline, durable replay bytes are stored elsewhere, Session/Event history retains an authorized reference, and a later replay can resolve the complete value.

Today the two load-bearing producers use the generic Artifact authority:

  • Read image snapshots, capped at 5 MiB;
  • Tool Result archives, rejected above 4 MiB.

Both consumers currently hydrate and verify the complete value. ArchiveRead applies offset/limit only after whole-object hydration; it is response pagination, not physical range I/O.

The current authority stores metadata in SQLite and payloads in the filesystem, then implements cross-domain publication, purge intent, stable-path recovery, global metadata reload/rewrite, and payload-tree inspection. On the motivating workspace, approximately 11.7k records / 6 GB are enough to block Host readiness for minutes because the access pattern is O(all records/files), and Session retirement can become O(M×N). That is a control-plane design failure, not a scale that should require a custom object store.

Decision

Use a dedicated SQLite database as the first physical substrate for capped, whole-object context offload.

Runtime consumers
  ├── ReadImageSnapshotStore
  └── ToolResultArchiveStore
            │ typed Session-owned references
            ▼
     ContextOffloadStore
            │ one SQLite transaction
            ▼
     context-offload.sqlite
       ├── context_blobs
       ├── context_refs
       └── usage counters

This database is separate from runtime.sqlite so multi-gigabyte replay payloads do not inflate operational-state backup, migration, or ordinary query paths. Blob bytes and context references live in the same database and commit atomically.

The current storage contract remains capped whole-object. This issue does not add streaming writes or claim true range reads. node:sqlite does not expose sqlite3_blob_open/read, and SQL substr(BLOB, ...) limits the returned value but does not guarantee bounded SQLite-side materialization.

If a future requirement needs genuinely large paged archives, evaluate fixed-size SQLite chunk rows with independently verifiable chunk integrity first. Add an external filesystem/S3-like CAS only after measured large-object or streaming workloads prove that chunked SQLite is insufficient.

Goals

  • Move new Read image snapshots and Tool Result archives off ArtifactStore.
  • Make blob + typed reference creation one SQLite transaction.
  • Preserve whole-object integrity verification and Session authorization.
  • Make Session copy copy references, not payload bytes.
  • Make Session retirement proportional to references owned by that Session.
  • Enforce explicit per-blob, per-Session logical, and workspace physical quotas.
  • Keep payload enumeration, reconciliation, GC, checkpoint, vacuum, and compaction off Host Ready.
  • Migrate still-referenced legacy replay payloads without rewriting immutable RuntimeEvents.
  • Stop producing unconsumed Write/Edit/Bash derivation Artifacts independently.

Non-goals

  • Redesigning Deep Research, upload, subagent write-back, or Desktop Artifact surfaces.
  • Moving Session recap effect state; it belongs in operational state and can be handled separately.
  • A generic replacement for every ArtifactSource.
  • Large-object streaming or physical range-read semantics.
  • Long-lived dual writes between ArtifactStore and the new context store.
  • Running full VACUUM, GC, or legacy cleanup before Host Ready.

Durable contracts

Core reference

Add a durable reference that names logical Session-owned context, not a filesystem path:

export interface SessionContextRef {
  readonly kind: 'session_context';
  readonly sessionId: string;
  readonly refId: string;
}

StorageRef accepts this variant for new Read image snapshots. Persisted session_file values remain a legacy input during the bounded cutover.

Tool Result placeholders move to a new rewrite version with contextRefId. The decoder continues to accept v1 artifactId placeholders. Migration preserves ref_id = legacy artifactId where possible, so old immutable RuntimeEvents resolve through the new store without event rewriting.

Storage authority

export type ContextOffloadOwner =
  | {
      readonly kind: 'read_image_snapshot';
      readonly ownerId: string;
    }
  | {
      readonly kind: 'tool_result_archive';
      readonly ownerId: string;
    };

export interface ContextOffloadRecord {
  readonly refId: string;
  readonly sessionId: string;
  readonly owner: ContextOffloadOwner;
  readonly blobId: string;       // canonical lowercase SHA-256
  readonly sizeBytes: number;
  readonly mediaType: string;
  readonly createdAt: number;
}

export type ContextOffloadPutResult =
  | { readonly ok: true; readonly record: ContextOffloadRecord }
  | {
      readonly ok: false;
      readonly reason:
        | 'too_large'
        | 'session_quota_exceeded'
        | 'workspace_quota_exceeded'
        | 'identity_conflict'
        | 'unavailable';
    };

export type ContextOffloadReadResult =
  | {
      readonly ok: true;
      readonly record: ContextOffloadRecord;
      readonly bytes: Uint8Array;
    }
  | {
      readonly ok: false;
      readonly reason:
        | 'not_found'
        | 'session_mismatch'
        | 'too_large'
        | 'corrupt'
        | 'unavailable';
    };

export interface ContextOffloadStore {
  put(input: {
    readonly sessionId: string;
    readonly owner: ContextOffloadOwner;
    readonly bytes: Uint8Array;
    readonly mediaType: string;
    readonly expectedSha256?: string;
  }): Promise<ContextOffloadPutResult>;

  read(input: {
    readonly sessionId: string;
    readonly refId: string;
    readonly maxBytes: number;
  }): Promise<ContextOffloadReadResult>;

  copyReferences(input: {
    readonly sourceSessionId: string;
    readonly targetSessionId: string;
    readonly references: readonly {
      readonly sourceRefId: string;
      readonly targetOwner: ContextOffloadOwner;
    }[];
  }): Promise<{
    readonly copied: readonly {
      readonly sourceRefId: string;
      readonly targetRefId: string;
    }[];
  }>;

  releaseReference(input: {
    readonly sessionId: string;
    readonly refId: string;
  }): Promise<void>;

  retireSession(sessionId: string): Promise<{
    readonly releasedReferences: number;
    readonly releasedLogicalBytes: number;
  }>;

  collectGarbage(input: {
    readonly olderThan: number;
    readonly maxBlobs: number;
    readonly maxBytes: number;
  }): Promise<{
    readonly deletedBlobs: number;
    readonly deletedBytes: number;
    readonly hasMore: boolean;
  }>;

  usage(sessionId?: string): Promise<{
    readonly references: number;
    readonly logicalBytes: number;
    readonly physicalBytes: number;
  }>;

  close(): void;
}

The public interface remains asynchronous even if the first implementation uses DatabaseSync, preserving the option to move storage work to a dedicated worker without changing consumers.

ownerId is stable within a Session. A retry with the same (sessionId, owner.kind, ownerId) and identical bytes returns the existing reference. Different bytes for the same identity fail with identity_conflict.

Typed consumer facades

The physical store does not expose ArtifactSource, product visibility, filenames, preview listing, or arbitrary retention classes.

export interface ReadImageSnapshotStore {
  snapshot(input: {
    readonly sessionId: string;
    readonly ownerId: string;
    readonly bytes: Uint8Array;
    readonly mimeType: string;
  }): Promise<SessionContextRef>;

  read(input: SessionContextRef): Promise<ContextOffloadReadResult>;
}

export interface ToolResultArchiveStore {
  archive(input: ToolResultArchiveRecorderInput): Promise<{
    readonly contextRefId: string;
  }>;

  readForReplay(input: ToolResultArchiveReaderInput): Promise<ToolResultArchiveReadResult>;

  readResource(input: ToolResultArchiveResourceReadInput): Promise<ToolResultArchiveReadResult>;
}

The Host derives both facades from the same authenticated, root-lease-bound ContextOffloadStore. The existing ToolResultArchiveCapability remains indivisible: writer, replay reader, ref reader, and ArchiveRead decoder are still wired together.

SQLite schema

PRAGMA journal_mode = WAL;
PRAGMA synchronous = FULL;
PRAGMA foreign_keys = ON;
-- Set before creating tables in a new database.
PRAGMA auto_vacuum = INCREMENTAL;

CREATE TABLE context_blobs (
  blob_id BLOB PRIMARY KEY CHECK(length(blob_id) = 32),
  payload BLOB NOT NULL,
  size_bytes INTEGER NOT NULL CHECK(size_bytes >= 0),
  created_at INTEGER NOT NULL CHECK(created_at >= 0)
);

CREATE TABLE context_refs (
  ref_id TEXT PRIMARY KEY,
  session_id TEXT NOT NULL,
  owner_kind TEXT NOT NULL CHECK(
    owner_kind IN ('read_image_snapshot', 'tool_result_archive')
  ),
  owner_id TEXT NOT NULL,
  blob_id BLOB NOT NULL REFERENCES context_blobs(blob_id) ON DELETE RESTRICT,
  media_type TEXT NOT NULL,
  created_at INTEGER NOT NULL CHECK(created_at >= 0),
  UNIQUE(session_id, owner_kind, owner_id)
);

CREATE INDEX context_refs_session
  ON context_refs(session_id, created_at, ref_id);

CREATE INDEX context_refs_blob
  ON context_refs(blob_id);

CREATE TABLE context_session_usage (
  session_id TEXT PRIMARY KEY,
  reference_count INTEGER NOT NULL CHECK(reference_count >= 0),
  logical_bytes INTEGER NOT NULL CHECK(logical_bytes >= 0)
);

CREATE TABLE context_store_usage (
  singleton INTEGER PRIMARY KEY CHECK(singleton = 1),
  blob_count INTEGER NOT NULL CHECK(blob_count >= 0),
  physical_bytes INTEGER NOT NULL CHECK(physical_bytes >= 0)
);

Usage counters update in the same transaction as blob/reference mutations. A Session quota counts logical referenced bytes; the workspace quota counts unique physical bytes, so Session copy cannot bypass logical quota and dedup does not double-count physical storage.

The per-kind hard limits remain 5 MiB for Read images and 4 MiB for Tool Result archives. Session/workspace quota values must be selected from retained-producer measurements and committed as explicit product constants before implementation is considered complete.

Write, read, copy, and delete semantics

Put

One BEGIN IMMEDIATE transaction:

  1. Validate the owner-specific byte limit and expected SHA-256.
  2. Resolve the stable ref identity and reject identity drift.
  3. Check Session logical and workspace physical quota counters.
  4. INSERT ... ON CONFLICT DO NOTHING for the content-addressed blob.
  5. Insert the typed reference.
  6. Update usage counters.
  7. Commit.

The store verifies size/hash on an existing blob identity before accepting the reference and fails closed on inconsistency.

Read

  1. Query by (session_id, ref_id); knowing a blob hash never grants access.
  2. Reject values beyond the caller and owner-kind cap before returning bytes.
  3. Materialize the complete capped BLOB.
  4. Verify byte length and SHA-256.
  5. Return the whole object.

ArchiveRead may continue slicing its model response, but storage does not advertise range-read semantics.

Session copy

Copy selected context_refs in one transaction and point them at the same immutable blob rows. Return an old-ref to new-ref map for RuntimeEvent/message rewriting. Physical byte count must not increase.

Release, retirement, and GC

Reference deletion is indexed and Session-scoped. It does not synchronously delete payload rows. Background GC deletes only blobs with no references, in bounded (maxBlobs, maxBytes) batches older than a watermark. Incremental vacuum and WAL checkpointing are maintenance work, never readiness work.

Replay-critical references have no independent TTL. Known failed writes perform best-effort releaseReference; a hard crash between context commit and RuntimeEvent commit may leave a safe orphan. The orphan counts against explicit quota and is removed on Session retirement. A later bounded reconciler may remove proven-unreferenced owner identities, but is not required for the first cutover and must never infer that old replay-critical data is disposable from age alone.

Failure and degraded behavior

Failure point Required behavior
Crash before context transaction commit No blob or reference becomes visible.
Retry same owner and same bytes Return the existing reference.
Retry same owner with different bytes Fail identity_conflict.
Context commit succeeds, RuntimeEvent commit fails Safe orphan; best-effort release, quota-bound until Session retirement.
RuntimeEvent references missing/corrupt context Scoped not_found/corrupt; never scan the payload set or silently substitute another Session.
Archive put fails Do not replace the original Tool Result with an archive placeholder.
Image snapshot put fails Return an explicit Read/storage failure; do not emit a dangling SessionContextRef.
Session mismatch Fail closed before returning bytes.
Quota/database full Typed failure; Host remains available and pruning stays inline where possible.
Context database cannot open Context capability is unavailable with diagnostics; unrelated Host readiness must not perform payload recovery scans.
GC/vacuum/checkpoint failure Report and retry out of band; do not poison Host Ready or future bounded maintenance.

Legacy transition

Do not rewrite immutable RuntimeEvents.

  1. New writes switch directly to SessionContextRef / Tool Result placeholder v2; no dual write.
  2. A post-Ready, versioned, resumable migration discovers durable legacy replay references from RuntimeEvents/messages, rather than migrating every record solely by ArtifactSource.
  3. For each referenced v1 Tool Result archive or Read image snapshot, copy and verify the payload in a bounded transaction. Preserve the legacy Artifact id as ref_id where possible.
  4. The compatibility reader resolves v1 archive placeholders and legacy snapshot session_file refs through the new context database, with legacy Artifact fallback only until that reference is migrated.
  5. Missing or malformed legacy payloads produce a recorded scoped degradation; they do not block unrelated Host readiness.
  6. Obsolete Write/Edit/Bash derivations are purged without migration.
  7. Reclaim old payloads in bounded post-Ready batches. Do not call the existing per-Session purge loop repeatedly.
  8. Remove the legacy context fallback immediately after migration evidence proves all load-bearing references are resolved. Other product-owned Artifact classes remain outside this issue.

The migration must expose progress and be restartable. Cleanup is never allowed to turn a one-time cutover into another O(M×N) startup path.

Implementation sequence

  1. Evidence and contract: measure count/total/p50/p95/p99/max for retained producers; freeze Session/workspace quotas and whole-object performance budgets.
  2. SQLite authority: add the dedicated owner, schema, atomic put/read/release, counters, bounded GC, backup, and corruption/error mapping.
  3. Typed consumers: route Read image snapshots and Tool Result archives through their facades; add SessionContextRef and placeholder v2 while retaining v1 decoding.
  4. Lifecycle: make conversation copy clone references only; make Session retirement release one Session's refs; run GC/checkpoint/incremental vacuum outside readiness.
  5. Legacy cutover: migrate durable referenced payloads post-Ready, preserve legacy ids, stop unused producers, and reclaim obsolete payloads in bounded batches.
  6. Retirement follow-through: remove legacy context paths from ArtifactStore; reassess perf(storage): artifact metadata rewrites the full record table on every mutation (O(M×N) during startup) #4037/perf(storage): batch multi-Session artifact purge during session retirement (O(M×N) guard scans) #4038 as fallback-only stopgaps and close fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027 only when a 10–12k legacy workspace no longer performs scale-dependent Artifact work before Ready.

Each slice must be independently revertible. No slice introduces long-lived dual writes or a second new physical authority.

Verification and acceptance invariants

Contract and correctness

  • Same owner/same bytes retry is idempotent; owner drift conflicts.
  • Blob and ref are never partially committed inside the context database.
  • Cross-Session reads fail even when the caller knows ref/blob identity.
  • Read image and archive hard byte caps are enforced before commit and before return.
  • Whole-object SHA-256 and byte length are verified.
  • Archive put failure leaves the original Tool Result unpruned.
  • Session copy creates new refs without copying payload bytes.
  • Session retirement touches only that Session's indexed refs.
  • GC cannot delete a referenced blob and obeys both batch limits.
  • Quota counters remain transactionally consistent across dedup, copy, release, retirement, and GC.

Crash and migration

  • Child-process crash tests cover before commit, after blob insertion, after ref insertion, after context commit/before RuntimeEvent commit, and during bounded GC.
  • Reopening relies on SQLite recovery; no payload enumeration is performed.
  • Legacy migration is idempotent and resumes from every batch boundary.
  • Unmigratable legacy content degrades explicitly without blocking unrelated Sessions.
  • No immutable RuntimeEvent is rewritten.

Performance

  • Benchmark monolithic SQLite BLOB put/get and peak RSS at retained-producer p50/p99/max on Linux, macOS, and Windows.
  • Use EXPLAIN QUERY PLAN evidence for read, copy, retire, quota, and GC queries.
  • A workspace with approximately 10–12k legacy records / 6 GB reaches Host Ready without enumerating, decoding, hashing, reconciling, vacuuming, or compacting the payload population.
  • Host Ready and unrelated Session retirement do not scale with total context blob/ref count.
  • WAL checkpoint, GC, legacy cleanup, and incremental vacuum execute only after Ready or in an explicitly surfaced bounded cutover step.

Alternatives considered

Patch current ArtifactStore only

Indexed delta writes (#4037) and batched purge (#4038) are valid stopgaps but preserve the cross-SQLite/filesystem authority and its mixed product taxonomy. They do not establish the context-offload boundary.

Filesystem immutable CAS + SQLite references

Valid for streaming/large objects, but reintroduces two durability domains, orphan policy, filesystem cleanup, and range integrity. Current retained consumers are capped whole-object values, so this complexity is not justified yet.

Chunked SQLite now

Provides bounded materialization for true paging, but requires chunk protocol, ordering, independently verifiable integrity, and more rows. Current consumers already hydrate at most 4–5 MiB whole objects. Defer until a new large-object contract requires it.

Put payloads in runtime.sqlite

Would permit wider single-database transactions but couples multi-gigabyte replay bytes to operational-state backup, schema migration, and ordinary runtime queries. A dedicated database gives the payload domain an independent capacity and maintenance boundary.

Relationships

AI use

Generative tooling made a substantive contribution to repository inspection, interface/schema design, failure analysis, migration planning, and this issue text. All claims and proposed contracts were reviewed against current main and the linked discussion.

Activity

  1. likun666661
    added
    enhancementNew feature or request
    on Aug 28, 2026
  2. likun666661
    self-assigned this
    on Aug 28, 2026

  3. M4n5ter commented on Aug 28, 2026

    @M4n5ter
    M4n5ter
    on Aug 28, 2026
    Member
    English

    This is a strong implementation plan for removing the scale-dependent ArtifactStore path from Host Ready, and I think the SQLite direction is plausible for capped Read image snapshots. Before implementation, however, I think we need to resolve one more architectural question: what is the single canonical byte owner of a Tool Result?

    Today the complete Tool Result is committed into runtime_events.payload_json, and pruning later serializes and writes the same result into an archive Artifact. As currently written, I cannot find a step in this issue that changes the T2 RuntimeEvent contract, so the proposed database appears to move that second copy from ArtifactStore into context-offload.sqlite.

    That leaves us in an ambiguous state:

    • If the archive is only a derived materialization of the immutable RuntimeEvent, it should be reconstructible and disposable. It should not need replay-critical retention, mandatory migration, or inclusion in a complete backup.
    • If the context body is replay-critical and independently authoritative, the RuntimeEvent should retain only a durable reference rather than another complete copy.

    I think we should explicitly choose one of these coherent targets:

    1. Ledger-canonical: Tool Result reads resolve the original RuntimeEvent through a typed facade; context-offload.sqlite owns Read image snapshots only.
    2. Context-canonical: large Tool Results are written once to the context store at the T2 boundary, and new RuntimeEvents persist only a durable context reference. Small results may remain inline.

    Read image snapshots are different: the workspace source can change, so the snapshot bytes genuinely need a separate canonical owner.

    A few related contracts also need adjustment before freezing the schema:

    1. 4 MiB is not currently a Tool Result storage limit. It is enforced by the ArchiveRead resource decoder, while the archive writer, replay reader, and MakaTool<R = unknown> contract can handle or produce larger results. Making 4 MiB a store hard limit would reject exactly the results that most need offloading and leave them inline. We should either enforce an upstream Tool Result limit at the execution boundary or support larger stored bodies, potentially with fixed-size chunk rows behind the same whole-object consumer API.

    2. The proposed GC is not yet provably bounded. maxBlobs and maxBytes bound deletion output, but the schema has no indexed representation of GC eligibility. An anti-join may inspect an unbounded number of referenced blobs before finding a small batch of unreferenced ones. A transactional ref_count/unreferenced_at state or an indexed context_gc_candidates table would make the query work itself bounded.

    3. An async TypeScript interface does not make DatabaseSync non-blocking. Node documents that all DatabaseSync APIs execute synchronously. We should freeze an event-loop stall budget and benchmark BLOB writes, reads, and checkpoints before deciding that a Worker is optional. If the budget is exceeded, the Worker must be part of the first authority implementation.

    4. The backup contract is currently contradictory. Keeping the context DB out of operational backup avoids multi-GB backup growth, but a restored RuntimeEvent/Session may then reference missing replay-critical bytes. Including it preserves replay but does inflate the backup. We should explicitly distinguish metadata-only backup from replay-complete backup, or include an ordered cross-database snapshot protocol and restore validation.

    5. “Independently revertible” needs an expand/contract definition for durable v2 refs. Once a writer emits v2 references, reverting to a binary without the v2 reader is unsafe. A safe rollout is reader-first, then writer activation; writer rollback may stop new v2 writes but must retain the v2 reader for the rollback window. This does not require long-lived dual writes.

    I would therefore keep the typed facades, quotas, Session-scoped authorization, post-Ready migration, and SQLite evidence work, but add a decision gate before schema implementation:

    Every model-visible value has exactly one canonical byte owner. Any second representation must be disposable and reconstructible.

    Once we decide whether Tool Result bytes are ledger-canonical or context-canonical, the remaining schema, migration, backup, and failure contracts become much easier to make internally consistent.

    中文

    这是一个很扎实的实施方案，能够把随 ArtifactStore 总规模增长的工作移出 Host Ready。对于有上限的 Read 图片快照，SQLite 方向也是合理的。不过在开始实现前，我认为还需要解决一个架构问题：Tool Result 的唯一字节权威到底是谁？

    目前完整 Tool Result 已经提交到 runtime_events.payload_json，pruning 随后又序列化同一个结果并写入 archive Artifact。按这个 issue 当前的描述，我没有看到修改 T2 RuntimeEvent 合同、停止保存完整结果的步骤，因此新数据库看起来只是把第二份副本从 ArtifactStore 搬到 context-offload.sqlite。

    这会留下一个不明确的中间状态：

    • 如果 archive 只是 immutable RuntimeEvent 的派生物化，它就应该可以重建和删除，不应需要 replay-critical retention、强制迁移或进入完整备份。
    • 如果 context body 是 replay-critical 的独立权威，那么 RuntimeEvent 应只保存 durable reference，而不是再保存一份完整内容。

    我认为应该明确选择下面两个自洽目标之一：

    1. Ledger 为权威： Tool Result 通过 typed facade 读取原始 RuntimeEvent；context-offload.sqlite 只负责 Read 图片快照。
    2. Context store 为权威： 大 Tool Result 在 T2 边界只写一次 context store，新 RuntimeEvent 只保存 durable context reference；小结果可以继续 inline。

    Read 图片快照与 Tool Result 不同：workspace 原文件可能变化，因此快照字节确实需要一个独立的 canonical owner。

    在冻结 schema 前，还有几个相关合同需要调整：

    1. 4 MiB 目前不是 Tool Result 的存储上限。 它只由 ArchiveRead resource decoder 执行，而 archive writer、replay reader 以及 MakaTool<R = unknown> 都可能处理或产生更大的结果。把 4 MiB 直接变成 store hard limit，会拒绝最需要 offload 的结果并让它们继续 inline。我们应当在执行边界正式限制 Tool Result 大小，或者支持更大的存储对象；内部可以使用 fixed-size chunk rows，同时继续向消费者提供 whole-object API。

    2. 当前 GC 还不能证明是真正有界的。 maxBlobs 和 maxBytes 只限制删除结果量，但 schema 没有可索引的 GC eligibility。anti-join 可能检查任意数量的仍被引用 blob，才能找到少量孤儿。可以事务性维护 ref_count/unreferenced_at，或者增加带索引的 context_gc_candidates 表，使查询工作量本身也有界。

    3. 异步 TypeScript 接口不会让 DatabaseSync 变成非阻塞。 Node 明确说明 DatabaseSync 的所有 API 都同步执行。应当先冻结 event-loop stall budget，并测试 BLOB 写入、读取和 checkpoint；如果超过预算，Worker 就必须属于首版 storage authority，而不是未来可选优化。

    4. Backup 合同目前存在矛盾。 context DB 不进入 operational backup，可以避免多 GB 备份增长，但 restore 后 RuntimeEvent/Session 可能引用缺失的 replay-critical bytes；如果包含它，则备份仍会膨胀。应明确区分 metadata-only backup 与 replay-complete backup，或者定义有顺序的跨数据库 snapshot 和 restore validation。

    5. 对于 durable v2 ref，“independently revertible”需要 expand/contract 定义。 writer 一旦产生 v2 reference，就不能安全回滚到不认识 v2 的旧 binary。安全顺序应当是先发布 reader，再启用 writer；回滚 writer 时可以停止新的 v2 写入，但在回滚窗口内必须保留 v2 reader。这不需要长期 dual writes。

    因此我建议保留 typed facades、quota、Session-scoped authorization、post-Ready migration 和 SQLite evidence 工作，但在 schema 实现前增加一个决策门：

    每个 model-visible value 必须只有一个 canonical byte owner。任何第二种 representation 都必须可以删除并从权威重建。

    确定 Tool Result 是 ledger-canonical 还是 context-canonical 之后，其余 schema、migration、backup 和 failure contracts 会更容易保持内部一致。

  4. Astro-Han
    removed
    enhancementNew feature or request
    on Aug 29, 2026
  5. Astro-Han
    added the
    Task
    issue type on Aug 29, 2026

  6. likun666661 commented on Sep 7, 2026

    @likun666661
    likun666661
    on Sep 7, 2026
    MemberAuthor

    Status reconciliation and proposed remaining scope

    Rechecked this issue and the canonical-owner review against main at b06eb02e6aecbc56e003439ce61b371baf0b4fc9 (2026-09-07). This is a proposed revision, not a claim that the original acceptance criteria have passed.

    Current implementation

    #4078, #4159, #4165, #4168, #4182 and #4184 are merged. Read images now use Session context refs, with SQLite metadata and content-addressed managed files. This differs from the original all-in-SQLite payload transaction; I would retain the shipped representation rather than move images again just to match the old proposal.

    Production limits are 5 MiB per Read image, 1 GiB logical per Session and 20 GiB physical per workspace; tool_result_archive remains disabled at zero bytes (composition). These constants do not substitute for measured capacity evidence.

    After #4287/#4350, archive serialization uses the effective durable model projection, not simply the raw execution result. New archive payloads still go to ArtifactStore. The 4 MiB cap is an ArchiveRead read limit, not a general Tool Result execution/storage limit.

    Gaps to address before extending the writer

    1. Retirement drains global GC. retireContextSession releases one Session's refs, then loops until global hasMore=false, with an effectively unlimited byte budget. Bounded batches do not make the whole call bounded.
    2. Cleanup still delays readiness. Recovery awaits reclaimUpgradeResidue. Catching cleanup failures does not prevent its full-population work from delaying Ready.
    3. Backup/export omits context data. Operational backup and Session bundle policy cover runtime.sqlite/artifacts, not the context database and managed values. Shipped image refs need a replay-complete restore contract.
    4. Artifact access remains partly global. Writes now use delta applyChanges, but load/reload still reads all records, and purge resolves non-target records for alias protection. The original full-table-write diagnosis is partly outdated; the remaining global reads/scans are not fixed merely by switching replay consumers.

    Proposed decision: ledger-canonical Tool Result archives

    I agree with the earlier review's canonical-owner decision gate. Given the durable projections now present, I recommend:

    • Retain execution facts and persisted model projections in their existing ledgers, with their distinct responsibilities.
    • Make new ArchiveRead refs resolve the exact projection version replaced by the archive transition, without publishing a second archive payload to ArtifactStore or context-offload.sqlite.
    • Keep non-reconstructible Read-image snapshots in the current Session-owned context store.
    • Leave uploads, Deep Research and product-owned child outputs outside this migration.

    This explicitly revises the issue's original two-producer storage decision.

    The resolver must bind Session, target event, source projection digest, predecessor identity where needed, archive encoding version, body hash and length. It must reconstruct the pre-replacement projection, not today's placeholder; it must not expose arbitrary raw execution evidence or rerun today's tool projector.

    Address one target with bounded source reads, not a whole-Session event/transition scan. Reuse the reducer's chain-selection rules; query indexes remain rebuildable, not a new authority. Freeze archive serialization by version and fail closed on hash/length mismatch. New readable refs must correspond to an accepted archive relationship, not just knowledge of an event id/hash.

    Suggested PR sequence

    PR Scope
    1. Readiness and maintenance Retirement releases only its own refs. Start GC/upgrade cleanup explicitly after Ready; bound candidates, bytes, file operations and scheduling slices. Retain removal fences and durable retry intent. Measure synchronous stalls rather than assuming async signatures make SQLite non-blocking.
    2. Backup/export integrity Include context refs and required managed bytes; validate restored reference closure and hashes. Prefer explicit offline/quiesced snapshots first, fencing writes, retirement and GC. The existing Artifact lock alone is insufficient. Version the format; do not silently label incomplete snapshots replay-complete.
    3. Ledger archive reader Add versioned, Session-authorized, bounded lookup and byte-exact reconstruction. Test source/predecessor validation, encoding stability, concurrent transition branches and cross-Session refusal. Reader-first; no writer activation.
    4. Writer and copy cutover Active/stale pruning uses ledger refs through the existing transition owner and stops writing archive payloads. Commit failure leaves the projection unchanged. Update live/restart/compaction/ArchiveRead and copy/branch remapping together.
    5. Legacy cutover and evidence Inventory durable reachability, classify old bodies, migrate snapshots and reclaim only proven redundant bytes. Complete large-workspace and cross-platform acceptance. Start benchmark fixtures in PR 1, not at the end.

    A writer rollback must retain the new reader once durable refs have been emitted; a compatibility epoch alone does not make an old binary safe to reopen the data.

    Legacy safety and acceptance

    • Reclaim an old archive only after reconstruction matches its exact hash/length and its old durable identity has a working compatibility route.
    • Preserve still-referenced bodies that cannot be reconstructed. Missing historical projections or encoding differences do not authorize guessing from raw results or deleting the only copy.
    • Migrate legacy Read snapshots with Session-scoped identity mappings; do not rewrite immutable RuntimeEvents.
    • Discover references across supported events, projections, transition replacements, legacy messages and saved resource identities. ArtifactSource alone is not reachability evidence.
    • Audit consumers before stopping Write/Edit/Bash derivations: tool_result still participates in shared/child-output surfaces despite being hidden from ordinary artifact listing.
    • Use resumable, budgeted migration, race-safe with Session retirement: publish/verify replacement, persist compatibility mapping, then delete old bytes.
    • Removing a legacy physical-store fallback and removing its durable-reference decoder are separate decisions. Define the supported upgrade window before either contraction.

    For closure, compare 0/1k/approximately 12k-record populations, including the motivating approximately 6 GiB workload, on Linux/macOS/Windows. Separate first schema upgrade from ordinary reopen; measure rows/files touched before Ready and by an unrelated Session retirement, latency, RSS and longest synchronous stall. Remaining product ArtifactStore scans must stay tracked in #4027/#4037/#4038 rather than being declared solved by this cutover.

    Verification scope: source inspection and issue/PR history only. No new builds, tests, retained-producer measurements, real-data migration or cross-platform benchmarks were run for this proposal.

    中文摘要

    建议修订剩余方向，不继续把 Tool Result archive 副本搬进 context SQLite：

    • Read 图片已上线，但退休后的全局 GC、Ready 阶段历史清理，以及 context 备份/导出缺失需要优先处理。
    • Tool Result 已有 durable projection，优先选择 ledger-canonical：ArchiveRead 精确重建归档时被替换的那一版 projection，不再写第二份 payload。
    • 按“后台维护 → 备份完整性 → ledger reader → writer/copy 切换 → legacy 分类与性能验收”分 PR。
    • 旧 archive 只有在重建 hash/长度完全一致且兼容引用可读后才能删除；不可重建且仍被引用的数据必须保留。
    • 单批有界不等于整个调用有界；存储切换完成也不代表 fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027 已通过验收。

    这是待确认的设计修订，本轮未执行新测试或真实数据迁移。

    AI disclosure: Codex assisted with repository inspection and drafting this proposal.

  7. Astro-Han
    added 2 commits that reference this issue on Sep 7, 2026

    fix(runtime-host): defer bounded storage maintenance until ready

    1edd125

    fix(storage): preserve context images in offline snapshots

    cde2423

  8. likun666661 commented on Sep 14, 2026

    @likun666661
    likun666661
    on Sep 14, 2026
    MemberAuthor

    Closure audit: not ready to close

    Rechecked current main (bf6e94229, 2026-09-14) against this issue's acceptance criteria.

    Substantial work has landed: context-offload and production Read images (#4078/#4159/#4165/#4168/#4182/#4184), Ready-safe maintenance (#4963), context-aware backup/export (#4965), ledger archive evidence and writer/copy integration (#4972/#4987), Session bundle portability and atomic context import (#5113/#5139/#5186/#5196/#5197), and unified bounded Read/tool-result pruning (#5247).

    The issue is nevertheless not complete:

    1. There is no resumable historical inventory/migration that classifies legacy tool_result_archive artifacts by ledger reachability and exact reconstruction.
    2. packages/runtime-host/src/server/execution-artifacts.ts still contains the legacy ArtifactStore archive reader, and tool_result_archive remains a compatibility source/schema. No proof-driven physical reclamation pass has removed historical bodies.
    3. No evidence proves the stated 10–12k-record / approximately 6 GiB workspace Ready and retirement acceptance. fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027, perf(storage): artifact metadata rewrites the full record table on every mutation (O(M×N) during startup) #4037 and perf(storage): batch multi-Session artifact purge during session retirement (O(M×N) guard scans) #4038 remain open.
    4. The Session bundle work solves portability and context payload integrity, but is not a legacy archive migration/reclamation pass.

    I recommend keeping #4071 open for a final legacy-retirement slice:

    • produce a read-only reachability/classification report;
    • prove exact ledger reconstruction by hash and length where possible;
    • retain non-reconstructible but referenced bodies;
    • reclaim only verified redundant payloads in bounded, resumable batches;
    • remove the legacy physical fallback only after migration evidence and an explicit upgrade window;
    • rerun the large-workspace Ready/retirement measurements.

    This is a source and issue-history audit only; no production data migration or deletion was run. Closing now would overstate completion.

  9. likun666661 commented on Sep 14, 2026

    @likun666661
    likun666661
    on Sep 14, 2026
    MemberAuthor

    Final closure decision

    After the subsequent implementation on main (#4987, #5113, #5139, #5186, #5196, #5197 and #5247), we are explicitly choosing the following scope for #4071:

    This makes the issue complete under the revised, explicitly loss-tolerant contract. Large-workspace ArtifactStore performance, old payload disk reclamation and any future compatibility cleanup should be tracked and closed independently with their own evidence. No production data migration or deletion is required for closing this issue.

  10. likun666661 commented on Sep 14, 2026

    @likun666661
    likun666661
    on Sep 14, 2026
    MemberAuthor

    Closing under the revised loss-tolerant contract recorded above. #4027, #4037 and #4038 remain independent ArtifactStore performance trackers.

  11. likun666661
    closed this as completedon Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Task

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions