[elee1766]

Summary

Refs #1994.

This is an experimental, opt-in session backend based on upstream master (a3ef562fc). The existing JSON snapshot + JSONL journal backend remains the default. No live session data or daemon was changed to gather these results.

• Add a SessionStorage interface and explicit-root SessionStore, with Files and SQLite implementations.
• Store metadata and ordered transcript vectors in SQLite transactions. Preserve JSON snapshot + adjacent journal import and self-contained JSON export, including unknown metadata.
• Route the picker, session search, resume lookup, crash recovery, replay, and /transcript export through the store for database-only sessions.
• Preserve imported file discovery ordering, missing-title display behavior, the picker debug budget, and saved-session tail.
• Include reproducible storage and real-picker benchmarks with result-equality gates. Only aggregate results are committed, never the private corpus or session metadata.

Measured workload

Frozen corpus: 3,018 sessions, 402 journals, 3.63 GB. All sessions imported without rejection and passed full persisted-value comparison. Every measured mutation was reopened and checked.

Both backends used the same unoptimized dev profile, rustc 1.98.1, warm filesystem cache, seven alternating repetitions, Intel i9-14900/Btrfs, and a shared host. These are prototype measurements, not release-build guarantees or total interactive jcode --resume latency.

Real picker

The production load_sessions() path ran in fresh processes, with external sessions disabled and scan limit 400. All runs returned 800 identical rows and displayed-metadata digests. Earlier runs with different selections or titles were rejected, fixed, and rerun.

Case	Files median / p95	SQLite median / p95
Database-only picker loading	3,583 / 4,359 ms	1,249 / 3,212 ms
Source JSON/journals retained after import	3,419 / 4,556 ms	1,146 / 8,131 ms

Median picker loading was 2.87x faster, or 2.98x with retained legacy files. The retained-file run had a slower SQLite tail observation. With seven samples on a shared host, this is not a claim of improved tail latency; p95 is the maximum of seven observations.

Full loads and writes

Sixteen size-quantile sessions, including the largest, sampled seven times. Full loads were timed on original imported state before retention preparation. Milliseconds:

Operation	Files median / p95	SQLite median / p95
Full load	2.10 / 1,097.03	2.05 / 31.23
Activity metadata commit	0.32 / 3.01	7.52 / 17.47
Title edit	3.61 / 13.89	3.37 / 7.24
Append	0.30 / 1.99	3.02 / 9.54
Truncate	6.70 / 513.33	14.21 / 600.55

The sample distribution intentionally covers size quantiles; its p95 is not a population-weighted percentile. SQLite's WAL synchronous=FULL commits have stronger crash durability than the current file writes, which do not fsync. Write latency is therefore not an equal-durability comparison. Activity commits, appends, and truncation medians are slower here.

Import took 167.15 seconds. Whole-run peak RSS was 1,712,536 KiB, including both backends and validation, not SQLite-only memory. Initial logical storage: files 3.376 GiB, SQLite 3.296 GiB including WAL/SHM. After mutations: files 3.031 GiB, SQLite 3.260 GiB. Neither backend is compressed.

Validation

• Final base session suite: 121 passed, 3 ignored.
• App-core session suite: 77 passed.
• SQLite search/replay: 2 passed.
• SQLite TUI checks: 4 passed, including real picker ordering/title parity, mixed native/legacy storage, and JSON transcript export.
• Full-corpus persisted-value parity: 3,018/3,018.
• Real-picker row and metadata parity: 800/800, seven repetitions each for database-only and retained-source layouts.
• Tests cover conflicting writers, metadata-only rebasing, unknown metadata, non-mutating import, corrupt import refusal, owner-only DB/WAL/SHM permissions, and schema migration.
• Benchmark examples compile and run; published machine-readable measurements were checked against private originals. No throwaway fixtures or corpus data are included. This is not a claim that the entire workspace test suite is green.

Scope and limitations

Please keep this draft while the storage boundary and remaining integrations are reviewed. SDK/harness APIs, cloud transfer, and auxiliary analytics still have direct-file consumers. Database-only sessions are not supported by every workflow yet.

There is no compression or FTS/trigram index in this PR. SQLite search uses the existing candidate/search path. Client and daemon processes must select the same backend. Native database records take precedence over legacy files; export before returning to a file-only binary.

File checkpoint retention behavior is unchanged. For steady-state writes the benchmark checkpoints the 16 file samples and gives both backends the same retained environment history before timing. Full import parity and untouched picker copies are checked before that preparation.

See docs/SESSION_STORAGE.md for the interface, reproduction commands, requirement-to-check mapping, and full tradeoffs. Aggregate evidence is in docs/benchmarks/session-storage-2026-10-27.json and docs/benchmarks/session-picker-2026-10-27.json.

[elee1766]

using fts5 for search isnt explored but probably should be