Repository navigation
design(atlas): semi-automated ingestion operations — source set, usefulness filter, disk/IO + plan-spend guardrails #6019
Description
Parent: #6018 (Atlas ingestion program). Statements S1–S5 of the 2026-11-04 brief.
Requirement (Drew, in substance)
Atlas updates become semi-automated on a long-term plan — not ad-hoc manual ingestion runs. The automation:
- Ingests the operational source set automatically: reboots / boot records, session logs, chat logs, statistics, and reports are added to Atlas as they are produced.
- Applies a usefulness filter: data that is useless to have in Atlas is NOT uploaded. Semi-automated ≠ indiscriminate — every source class carries an inclusion test (what question would this ever answer?).
- Monitors disk and I/O: ingestion watches storage consumption and I/O load on the source and target machines and throttles / pauses / reports rather than filling a disk or swamping a host.
- Never spends more than the plan: the current paid plan is a hard ceiling (today: Turso scaler, $24.92/mo yearly). Design treats plan limits (storage, reads/writes, DB count) as the budget envelope; growth that would cross the plan is surfaced to Drew, not silently incurred. Cf. #5468 (pm-stack mirror storage surge) for what unmonitored growth looks like.
Current state
- Ingestion today is script-driven and manual (scripts migrating into code + fleet under #5670); staging tier with merge-on-close promotion exists (#5669).
- Sources produce data continuously (boot records in
~/logs/init.jsonl+ init-status; OpenCode session store; Muse chats; quota/statistical reports) but nothing sweeps them into Atlas on a cadence. - Storage rulings (2026-11-04): Turso + D1 both used, Turso embedded replicas first-class, huge atlases shard across both, cold tiers → rustfs; TursoDB for all NEW databases; production atlas stays libSQL for now.
Design must answer
- Trigger model per source class: scheduled sweep vs event-driven (file close, session end, boot completion) vs hybrid; what "semi" means — which steps stay human-approved (new source classes? classification changes? bulk backfills?).
- The usefulness test: per-class inclusion rules, who/what applies them (static allowlist, size/shape heuristics, classifier pass), and how a rejected class is logged (rejection is data — cf. S32's unassociated-log principle).
- Disk/IO guardrails: watermarks, backpressure behavior, where the monitors live (source host vs ingest host), interaction with the delta-only design (companion issue under #6018) so sweeps stay cheap.
- Cost accounting: per-run and per-source ingest cost (storage delta, write volume) reported against plan headroom; alert before the ceiling, not after.
- Failure posture: a failed/partial sweep never corrupts or double-ingests (idempotent, content-addressed landing per #3125).
Done looks like
A design doc (in code, doc/) + follow-on implementation issues: per-source trigger table, usefulness rules, guardrail thresholds, cost report format, and the runbook for the recurring sweeps. Implementation itself is follow-on work, not this issue.
Activity
developing-today commentedon Nov 5, 2026
CODE-1685
dezren39 commentedon Nov 5, 2026
Research cost model + two structural rules for this design
Cluster 7 numbers, and two rules that shape the operations design:
(chunks_new, chunks_reused, tokens, model, $); reuse <90% on an append-mostly source means the delta protocol is broken, not the budget (ledger design: #6058).(path, size, mtime, head-hash)decides skips without opening files (the 21 GB rule: discovering the delta must not read the 21 GB), with sampled full-verification passes so mtime lies get caught. That protocol lives in the delta design (#6028 / research item 37, mapped there).Wave-2 children filed under this issue
— Wave-2 research fold-in, 2026-11-04 (prior-art doc:
~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md)dezren39 commentedon Nov 5, 2026
🤖 Agent note · done
Triage — task 10386, issue #6019
Verified fresh this shift: issue #6019 OPEN, labels
task/triage,area/infra,kind/design— no hard-stop labels. Parent programme issue #6018 OPEN. Cost-risk reference issue #5468 (Turso mirror storage surge) still OPEN, so the plan-ceiling guardrail in this design is live, not theoretical.Routing: this is a design ticket, not implementation. Its own "Done looks like" is a design doc in
doc/plus follow-on implementation issues (per-source trigger table, usefulness rules, guardrail thresholds, cost report format, sweep runbook). Dependencies to sequence against: ingestion scripts migration issue #5670, staging/promotion issue #5669, content-addressed landing issue #3125, and the delta-only companion design under issue #6018. No code fix is scoped before that design lands; no small_fix enqueued. Issue #6019 stays OPEN under issue #6018.Outcome for task 10386: triaged.