Repository navigation
design(atlas): staging-shard ingest gate — write → classify → dedup → provenance-check → publish #6057
Description
Parent: #6019 (semi-automated ingestion operations). Research source: prior-art doc ~/workspace/scratch/atlas-ingestion-brief-2026-11-04/prior-art-research.md (2026-11-04). Follow-up 38 (Cluster 7). Builds on the staging tier #5669 (merge-on-close promotion exists today).
Finding (research, in substance)
lakeFS's write-audit-publish pattern maps exactly onto Atlas's needs: ingest onto a staging branch/shard, validate, publish to the live set only if checks pass. The gate sequence per batch:
- Write to a staging shard (never directly to the searchable corpus).
- Classify — classification scan (family under task(atlas): classification revalidation scan — verify nothing sensitive sits at the wrong tier #6021) assigns/verifies tier before anything lands.
- Dedup — exact + near-duplicate edges recorded (ingest-time dedup pipeline, Atlas search: Jaccard as a third signal (rescore + ingest dedup); SPLADE as a triggered follow-on #5733 family).
- Provenance-check — every chunk carries a parseable source mark (sibling grammar design under atlas ingestion program: semi-automated updates, versioning, provenance, feedback — long-term design + tasks (2026-11-04 brief) #6018) + attestation (sibling under design(atlas): semi-automated ingestion operations — source set, usefulness filter, disk/IO + plan-spend guardrails #6019).
- Publish — promote to the live shard set; a failed batch never touches the searchable corpus, and retry is safe (content-addressed landing per atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125).
Done looks like
A gate design: stage contracts, per-stage pass/fail semantics, failure/quarantine handling, publish atomicity against fan-out search (#5755).
Activity
developing-today commentedon Nov 5, 2026
CODE-1722
dezren39 commentedon Nov 5, 2026
🤖 Agent note · done (triage)
Triage findings for issue #6057 (task 10423)
Verified current (2026-11-05, fresh
ghreads): issue #6057 is OPEN, filed 2026-11-05T05:27:46Z, carrying onlytask/triage+queue/queued+area/infra+kind/designat claim time — no hard-stop labels (held,user/owned,do-not-merge,do-not-auto-edit), no competing PR or branch, and no other issue references issue #6057 yet (search of6057 in:bodyreturns nothing), so no duplicate or superseding work exists.Routing: gate design whose stage contracts are defined by still-open sibling work. The five gate stages are each another family's contract, and every one of those families is still open (all verified fresh today):
status/needs-triage, queued): the foundation exists today (merge-on-close promotion), but its versioned schema/registry design has not been triaged yet; the gate's write contract lands on it.work/assigned) and attestation a sibling under parent issue design(atlas): semi-automated ingestion operations — source set, usefulness filter, disk/IO + plan-spend guardrails #6019 (OPEN,kind/design); the chunk provenance registry is issue atlas: provenance registry (chunk occurrences), content-addressed extracts, multi-DB vector resolution #3125 (OPEN,priority/high).Ordering recorded: the gate design pass on issue #6057 runs once the stage contracts it composes are at least drafted — staging tier issue #5669 triaged, classification semantics from issue #6021, provenance grammar/attestation from the issue #6018 / issue #6019 family — and its "done" deliverable stays as the body specifies (stage contracts, per-stage pass/fail semantics, failure/quarantine handling, publish atomicity against issue #5755). Drafting the gate before those contracts exist would invent per-stage semantics the sibling families are charged with deciding. No implementation issue is filed in this triage, and no code change is made; the issue stays open, labelled triaged + waiting-dependency to mirror this ordering.