experiment: learn state selection, retrieval routing, and observe-vs-act from verified trajectories #24

Agent priority — 2026-09-29 work audit

P1 after the State Packet cohort is frozen.

The first learned-policy experiment should use the smallest bounded decision family:

ACT | OBSERVE_MORE | ESCALATE

Input comes from the frozen State Packet/current-work cohort, not from synthetic labels invented for this issue.

Required order:

1. deterministic/rule baseline;
2. frozen grouped split by project/work lineage;
3. one simple statistical baseline;
4. one small learned selector in shadow-only mode;
5. system-cost + verified-transition metrics;
6. promotion/rollback decision.

Upstream runtime: kvnloo/z0intelligence#22
Frozen eval: kvnloo/z0evals#56

Do not give a learned selector authority over illegal transitions, and do not treat teacher/JEV agreement as ground truth.

Goal

Use verified z0intelligence trajectories to learn small, fast policies for navigating the intelligence stack.

Do not train a general replacement agent.

Target narrow functions that reduce expensive frontier-model work:

• query/facet routing
• entity linking
• ontology-slice activation
• state relevance selection
• blocking-unknown scoring
• observe-vs-act selection
• cache/state-packet selection
• verification anomaly detection

Training record

The upstream runtime should emit replayable trajectories:

intent
selected state / State Packet
candidate claims
unknowns
retrieved evidence refs
retrieval cost
action / transition
outcome
independent verification
later correction / contradiction

The key target is not "retrieval relevance."

It is:

Did the selected state contain the minimum sufficient information for a correct verified transition?

Baselines

Always compare learned approaches against:

• deterministic/rule selector
• lexical/metadata routing
• simple ranking/logistic/ridge model
• small MLP where appropriate
• frontier teacher

JEV/OpenJev may score a bounded legal action set, but teacher agreement alone is not ground truth.

Eval groups

Prevent leakage across related conversations/work:

event
-> trace/session
-> attempt lineage
-> project/work item
-> task family
-> harness/environment

Hold out complete projects/task families where feasible.

Metrics

Measure both decision quality and systems cost:

• verified transition success
• missed blocking unknowns
• false-block rate
• correction/retry rate
• abstention/risk coverage
• raw documents opened
• retrieval calls
• tokens
• latency
• cache hit/select accuracy
• frontier-model calls avoided

First experiment

Start with one bounded decision family from the State Compiler:

ACT
OBSERVE_MORE
ESCALATE

Inputs are a compact semantic state + bounded legal options.

Run learned policies shadow-only.

Distillation path

If the data supports it, progressively test:

frontier teacher
-> local SLM
-> tiny specialist
-> JEV-style scorer / rules

Promote per question family only.

Acceptance

• stable versioned trajectory schema
• frozen grouped train/eval split
• deterministic baseline
• one small learned selector tested shadow-only
• outcomes joined independently from model labels
• system-cost metrics reported alongside predictive metrics
• no learned model receives authority over illegal transitions
• clear promotion/rollback criteria