eval: tool-calling SLM tournament and Agent0-style self-evolution track

CURRENT PRIORITY — one useful history-derived skill, actually used (2026-10-10)

2026-10-23 source-discovery correction (no skill benefit claimed)

New evidence, not an adoption: z0intelligence PR #194 reports a real authorized read through the existing AgentsView exact session/message API, in addition to synthetic identity/freshness checks. It is a draft read-only facade, not installed ChatGPT memory, complete work-item provenance, or a newly required MCP deployment. PR #197 captures only explicit native Hermes JSON decision/correction records and has no live activation; z0evals PR #115 freezes one agent-policy → user-correction variant with retrieval/use NOT_RUN. None establishes two complete eligible learning episodes or any generated-skill improvement.

Critical-path simplification: At the next authorized local run, use AgentsView's already-existing native read API directly as a bounded, read-only source-discovery option, alongside already-working compatible extractors. Locate two distinct complete work items containing the original failure, correction and independently checkable resolution. Verify exact native lineage and authorization before invoking the unchanged ai-data-extraction corpus_to_skills.py on a minimal, properly supported input. Do not merge/activate #194 or #197, build a new reader/index/adapter, or treat #115 as an EL-23 success merely to satisfy this step. If the existing inputs cannot be qualified, report the bounded coverage gap and stop rather than invent examples. Isolate all inherited agent home/config environment to prevent repeat private-history exports. After input qualification, keep the existing first explicit-use and no-skill/manual-control test exactly as specified below. Latest known EL-23 pilot still has no actual skill-use/benefit result.

Deliver behavior before expanding the experiment platform. Kevin's requested outcome is that an agent benefits from prior experience on its next real z0intelligence task. A generated file, producer tests, training adapter, or supervisor does not deliver that outcome by itself.

Evidence checked: PR #35 ran the original corpus_to_skills.py on local Qwen2.5-Coder 7B and produced two uninstalled draft skills. The sampler consumed one record from nine synthetic qualification sessions. z0evals PR #95's native diagnostic explicitly supplied one proposal to local Qwen 7B: baseline 0/2, candidate 0/2, neither made an inspection call. This is a real negative for that experiment, not proof that history-derived procedures cannot help the working frontier harnesses. No verified gain or installed learning loop is established.

Smallest next experiment — existing tools, no new framework

Pick ONE recurring z0intelligence failure with an already-known correct procedure: for example, answering/implementing before inspecting the current owning source and tests. Use a few complete, already-authorized local episodes containing the correction and verified resolution. Do not truncate a giant corpus into an arbitrary fragment or bulk-export personal history.
Use the existing ai-data-extraction fork and its generator directly to produce ONE concise task-scoped candidate. Inspect existing drafts first; reuse one only if it matches this failure. Preserve native-source provenance privately. A short hand-authored procedure is the cheap control, not a substitute falsely credited to the generator.
Use an existing operational harness/model and its native skill loader in an isolated test profile. First explicitly invoke the skill and confirm actual delivery plus required source/tool inspection. Test automatic triggering only after explicit use works. If the harness cannot inspect the source, repair that single invocation/tool-access gap before evaluating skill quality.
Run a small matched comparison on three new task variations with the same source, model route, permissions and checker: no skill versus generated skill; compare the short manual control when needed to determine whether generation adds value. Check actual answers/artifacts with existing owning tests/z0evals. Record correctness, unnecessary calls and total available token/time measurements; include wrong/irrelevant-skill control. This is a pilot, not a statistical or sealed-certification claim.
Report one decision: keep for a reviewed task-scoped canary, revise, or discard. A positive canary must be observed on a subsequent real task; merely loading the file is not benefit. No global skill installation, production route change, paid experimental route, or authority expansion is implied.

Do not put model-weight training / EL #31, Groq-Cerebras admission, full repo reconciliation, or indefinite host supervision on this pilot's critical path. They are separate work. Fix only a prerequisite that prevents the experiment above from executing. Preserve their active leases and previous negative evidence; do not create competing writers.

Reuse native capabilities: Hermes already documents skill creation/loading, external skill directories and usage counters; Claude Code and Codex document task-scoped skills and explicit invocation. Check the installed build rather than inventing another loader or enabling broad background curation. Sources: Hermes skills, Claude Code skills, Codex skills.

Compact steering record: current user-visible failure | next smallest experiment | owner/active claim | exact candidate + observed consumer | comparison result | next decision. Keep the current decision here; detailed results stay on existing PRs. Only sanitized derived requirements, public code refs and aggregate results belong in public Git. Private transcripts, draft bodies, local paths and sensitive artifact identifiers remain private. Do not assume a hash makes private content anonymous.

Proposed division: Hermes produces the one candidate using the library; Codex independently checks new task outputs; Claude supplies already-working source/tool invocation context only if needed. These are proposed roles, not claims that new workers started. This priority correction preserves original author/library credit and the longer-term research below.

Longer-term research (not prerequisites for the pilot above)

Goal

Extend the existing observer-first evaluation work in #20 into a reproducible tournament for local tool-calling/orchestration policies, then test whether Agent0-style self-evolution can improve the winning local policy on our own traces.

Runtime inventory/control belongs in kvnloo/z0intelligence#20. Evolution Lab owns grouped experiments, frozen evaluation, training/search, and promotion evidence.

Candidates

Evaluate at least:

Nemotron-Orchestrator-8B
Qwen3.5-9B
Qwen3.5-4B
Hammer2.1-3B
Hammer2.1-7B where useful
FunctionGemma-270M after task-specific fine-tuning
JEV/OpenJev bounded scorer
deterministic/rules/logistic controls

Agent0 is a separate training methodology arm, not just another checkpoint.

Benchmark the actual control problem

Do not reduce this to JSON validity or one function-calling leaderboard.

For frozen work-item groups, measure:

verified task success
correct tool/model selection
invalid call rate
unnecessary call rate
dependency/order violations
tool irrelevance rejection
parallelization quality
retry count
recovery success
escalation precision/recall
stop/continue quality
wall-clock latency
TTFT
decision latency
GPU-ms
VRAM peak
tokens
dollar cost
calibration / abstention

Keep syntax/function-call accuracy and trajectory/control success as separate axes.

A/B matrix

Start with:

A. Qwen3.5-9B alone
B. Nemotron-Orchestrator-8B alone
C. deterministic legal-action compiler + orchestrator
D. compiler + JEV + orchestrator
E. compiler + JEV + orchestrator + Qwen fallback
F. same stack + distilled fly/mushroom/tiny specialist fast paths

Run 4B/3B/270M substitutions where the role permits.

Agent0-style track

Use verified traces to explore curriculum/executor co-evolution:

generate harder tool-aware tasks from real failure clusters
train executor policies against frozen verifiers
prevent self-generated reward from becoming ground truth
compare against DAgger/current autoresearch baselines
export only candidates that clear frozen held-out gates

No online self-modification of production authority.

Leakage / promotion

Preserve the grouping discipline from #20: retries, branches, continuations and repeated observations from one work item stay in one fold. Promotion is per role/question family, not “one best model overall.”

Acceptance

frozen local-tool-calling suite exists
trajectory metrics are distinct from schema/syntax metrics
3080 Ti latency/VRAM measurements are captured
deterministic/JEV/SLM compositions are compared, not only individual checkpoints
cost-quality-latency Pareto frontier is exported
Agent0-style training arm uses independent verifiers and held-out evaluation
promotion evidence can be consumed by z0intelligence
no security/capability authority is learned

Related: #20, kvnloo/z0intelligence#20

Free-tier spillover can become useful experiment capacity once the protected routing path is in place.

Kerdoios tracker: kvnloo/kerdoios#51

When real DSH/Hermes workload is insufficient to consume compatible Groq/Cerebras free quota before reset, Evolution Lab may offer explicitly queued eval/distillation/search work as low-priority fill. This must not generate meaningless filler solely to exhaust quota.

Keep the existing split discipline: protected confirm/OOD cases remain certification-only; background free-quota work should use training/validation or explicitly disposable research tasks.

Cross-linking a concrete implementation substrate for this track: #31.

Rather than building a second trainer inside Evolution Lab, #31 proposes treating SouthpawIN/evolutionary-training (via kvnloo/evolutionary-training) as an external Hermes-managed training provider. Evolution Lab keeps experiment/search/selection authority; the existing Gym Trainer + SFT/GRPO/Darwin/CMA-ES scripts do the training work; pinned z0eval contracts stay outside the evolvable surface; frozen evidence exports to z0evals only after revisions are pinned. That gives the Agent0-style arm here a reusable provider boundary instead of another one-off training loop. Credit/provenance remains with SouthpawIN for the Gym Trainer and training/evolution machinery.

Ownership correction / concrete next experiment (2026-10-09): the conversation-history → reusable skills/capability profiles → curriculum/evolution experiment belongs in Evolution Lab, NOT inside z0intelligence. Use the actual standalone fork https://github.com/kvnloo/ai-data-extraction and its existing extract_omp.py, extract_claude_code.py, extract_codex.py, and corpus_to_skills.py directly from the separate checkout, with private filtered outputs and a locally served OpenAI-compatible model. Do not reimplement/vendor extractor or generator code. ctx remains source-history search/provenance; compare coverage and keep existing indexed data authoritative for retrieval.

Proposed first frozen experiment: (A) collect a small private bounded OMP/Claude/Codex corpus using the original scripts, preserving branch/tool/diff identities; (B) run the original skill generator to produce review-only SKILL.md proposals; (C) evaluate no-skill baseline vs proposed skills on held-out tasks and compare independent correctness, bad/unnecessary tool calls, time/tokens/cost, leakage and regression. Evolution Lab owns candidate selection/curriculum/experiments; z0evals#93 and #92 own independent verification; z0intelligence#144 is only the downstream runtime/ctx/tool-selection feedback contract. No automatic installation, training or production routing before independently verified adoption. Give credit to the external project's contributors (the README credits Joshua Warren's OMP coverage and Umang Bhalla's Droid contribution among others); don't claim their work as our implementation.

Git factory board (non-default planning branch): https://github.com/kvnloo/z0intelligence/blob/coord/multiharness-strategy-board-20261009/docs/multiharness-factory-board.md . This is coordination/planning, NOT a completed extraction or outcome. Upstream library has no explicit license shown in current GitHub metadata: keep it separate and assess rights before redistribution; no need to copy its code to use the executable.

CLAIM — EL-23-X

task_id: EL-23-X (run the existing OMP extractor on bounded local inputs; compare coverage with ctx)
harness: Claude Code
owner: Evolution Lab (experiment); this lane executes only
branch: none in this repo yet; receipts and synthetic comparison notes only. No extractor code is copied or modified.
source_refs: kvnloo/ai-data-extraction at its current main, run unchanged from a separate private checkout; board coord/multiharness-strategy-board-20261009
worktree: private checkout and outputs under ~/.z0int/research/evolution-lab-23/ (mode 700, not in any public tree)
inputs: bounded to the isolated OMP qualification sessions from z0intelligence#137 (synthetic task, no personal history); the broad session store is not exported
blocked_by: nothing
Does not overlap EL-23-S (skill generation, Hermes) or ZE-93-V (Codex).

READY — EL-23-X (Claude Code lane): existing OMP extractor run unchanged; coverage compared with ctx

Extractor: kvnloo/ai-data-extraction @ 9176c2dd81c8a101b52debd4cec88573284e2b30, separate private checkout, zero local modifications. python3 extract_omp.py and python3 extract_omp.py --all-branches, both exit 0.
Inputs (bounded): 9 isolated OMP qualification sessions from z0intelligence#137 (synthetic task, 3.7 MB). The extractor also discovers ~/.omp by default; it was run with a staged home so the broad session store was not read.
ctx: 2.2.8, same 9 files imported into an isolated data root (--provider pi).
Outputs: private only (extracted_data/, comparison notes, receipt). Nothing from the corpus is in Git.

	Native records	Extractor	ctx
Sessions	9	9 conversations (10 with `--all-branches`)	9
Assistant messages	216	215 active branch / all with `--all-branches`	all, as events
Tool calls	337	337	337, grouped into 206 events (one per assistant message)
Tool results / errors	336 / 96	336 / 96	336
Thinking blocks	215	kept as `reasoning`	kept
Per-entry id and parent id	960 entries	dropped (session id, leaf id and source file only)	951 of 960 preserved
Custom session entries	363	8 kept (`custom_message` only)	kept as notices with data
Request timing (`duration`, `ttft`, `responseId`)	215 messages	dropped	kept
`reasoningTokens`, cost	62 / 216 messages	dropped	kept
Tool-result `details` (structured output, file ranges)	240 results	dropped	kept

What this means for the experiment

The extractor is a faithful conversation export: every message, tool call, result, error flag and reasoning block on the chosen branch. That is what corpus_to_skills.py consumes.
It is not a provenance source. Message-level identity, timing, reasoning-token and cost accounting, and structured tool output are gone, so anything that needs to cite a source event or measure cost has to join back to ctx by session id.
--all-branches exports each leaf as a full conversation. The one branched session produced two conversations of 53 and 59 messages sharing an identical 52-message prefix. For ZE-93-V that is a leakage path: hold-out has to be by session, not by exported conversation.
ctx's status line reports "Tool calls 206" for these sessions; that counts assistant messages containing calls, not the 337 calls.

Negative evidence

An isolated ctx import fails on this host while ~/.ctx is a symlink (daemon coordination … Not a directory); it worked with a private HOME. The host's own ctx status currently reports "Local usage configuration could not be read". ctx is not a dependable consumer here until that is fixed.
Running the extractor with Python's isolated mode (-I) breaks its sibling import; it needs the script directory on the path.
Bounded to 9 synthetic-task sessions with no subagents and no compaction; subagent and compaction coverage is untested.

Not done: no skill generation (EL-23-S), no extraction from personal history, no change to ctx's index or the user's ctx installation. Next dependency: EL-23-S can consume the private JSONL; ZE-93-V should take the branch-prefix finding.

READY — EL-23-X addendum: the existing Claude Code extractor on two synthetic workstream sessions

Same checkout (9176c2d, unmodified), python3 extract_claude_code.py, exit 0. Inputs: the two Claude Code consumer sessions from the z0intelligence#137 workstream (synthetic fixture, 51 native entries). Outputs private.

	Native	Extractor
Sessions	2	2 conversations, 10 messages
Assistant text / `tool_use`	2 / 3	kept
Tool results	3	exported as user messages; 0 structured tool results
Thinking blocks	3	dropped
Usage on assistant entries	8	dropped
`uuid` / `parentUuid`	35 entries	dropped
Attachments	20	dropped

Consequences for skill mining

Tool output arrives labelled as a user turn. A miner reading roles will treat command output as user intent.
Hook-delivered context is an attachment, so it is absent from the export. In both sessions the model answered from workstream memory injected by a hook; the exported conversation shows a correct answer to a prompt that does not contain the requirement. A skill inferred from that would credit the model with knowledge it was handed.
No message-level identity, so candidates cannot cite source events without joining back to ctx.

Discovery defects (negative evidence, extractor unchanged)

The home-directory patterns (~/.claude, …) are only checked inside the loop over ~/.config and ~/.local/share; with neither present it reports "No Claude Code installations found".
A non-default config directory (this host uses ~/.claude-home) is not discovered at all, so a default run here would silently export nothing from the real history.

Bounded to two short sessions with no subagents; not a statement about large sessions. Next dependency: EL-23-S should treat Claude Code exports as lossy on tool results and injected context, or draw those sessions from ctx.

CLAIM — EL-23-S (Hermes skill/capability mining; human-assigned by Kevin)

Owner/harness: Hermes / gpt-6.1-sol; native Codex subscription worker may execute isolated coding slices under this Hermes claim. This is NOT the existing Codex ZE-93-V lane.
Branch: exp/23-hermes-skill-mining-20261010, from Evolution Lab nightly 0212fee31a78cdba506f4d2684813962bfe0efb0; PR target preview, draft, no merge/automerge.
Lease: 2026-10-10T02:21:07.666027+00:00 through 2026-10-10T05:51:07.666027+00:00. Scope: EL-23-S generator execution, private candidate lineage/audit, experiment protocol and independently verifiable handoff. Planned source paths only docs/experiments/conversation-skills*, scripts/conversation_skills*, tests/test_conversation_skills* (or an existing equivalent after inspection). No extractor, generator, trainer, gold judge, production consumer, AGENTS, CI or routing rewrites.

Dependency-aware 3.5-hour work budget (not a claim that it has run):

Recover Claude EL-23-X frozen synthetic qualification corpus and original fork refs; run the unchanged fork generator against a confirmed loopback Ollama model with bounded input/output. Raw exports/draft skills stay private, no installation. If coverage/privacy/local-model gates fail, retain negative evidence.
Freeze candidate/source hashes, preserve branch/tool/diff limitations, test a minimal Evolution Lab experiment-side lineage/eligibility packet and provenance-only controls. Fail-then-pass for new behavior; unit + gym-smoke receipts bound to every published head.
Hand the frozen research packet to z0evals#93 / existing Codex owner for independent baseline-vs-skill qualification. Do not edit PR#94 or the independent gold contract, and do not self-certify improvement.
Refresh board, issue claims, PRs and test state after each verified slice. Select another eligible EL-23-S experiment dependency or independently scoped neutral latency/accounting audit if blocked; preserve Claude a90e3be two-upstream correction and Codex 199a849 reconciliation. Stop at the wall/session/subscription limit or when all safe nonoverlapping tasks are exhausted.

Non-overlap: Claude owns EL-23-X and OMP runtime/neutral bench; existing Codex owns ZE-93-V and judge qualification. Their branches, worktrees, receipts and private outputs remain unchanged. This lane only consumes frozen inputs read-only. ctx remains authoritative source search, with no new index/history database.

Spending/approval boundaries: coordinator uses existing ChatGPT subscription only; experiment inference is local-only, no API-key fallback or paid inference, no fine-tuning/download without a new explicit reason/approval. No production/default merges, no automatic skill installation, no CI approval bypass. Attribution and uncertain third-party redistribution rights retained. Runtime and quota failures are blockers, not success.

READY — EL-23-S slice 1 (Hermes): original local generator exercised; private proposals, no efficacy claim

Draft PR35: #35
Head 9a718a8b549f38c9bce16fb9cb36cad71f39d14e, base 0212fee31a78cdba506f4d2684813962bfe0efb0.

Consumed Claude EL-23-X's frozen active-branch export read-only: 9 isolated synthetic-task records, 2304611 bytes; seed23/12000 characters actually samples 1 record. Invoked the existing fork 9176c2d unchanged at a confirmed local Ollama endpoint. qwen2.5:3b exited1 (invalid JSON), 0 valid files. qwen2.5-coder:7b exited0, exactly 2 shape-valid SKILL.md proposals. Both raw corpus and draft text remain private and uninstalled; no personal-history export, paid experiment inference or generator/extractor copy.

Exact-head CPU unit: 271 tests, 1 skip; gym-smoke n8 and diff-check exit0. Original fork: 22 tests pass. These are plumbing checks, NOT candidate correctness, held-out gain, KEEP, savings or privacy approval. Missing event identity/timing/usage and branch-prefix leakage findings from EL-23-X remain authoritative; join back to native/ctx, never invent data.

Target correction: rollout says feature_target=preview, but preview shares no Git ancestor with the existing nightly source (GitHub rejected creation). Existing overnight_target=nightly has the verified ancestor, so this long unattended PR targets nightly as DRAFT. Its automerge job explicitly excludes drafts; readback shows draft=true and autoMergeRequest=null. No merge or history rewrite occurred.

Queue refreshed: Codex owns ZE-93-V/PR95 and is building the independent paired-baseline gate. Next producer slices: exact candidate/source identity and experimental eligibility/negative controls, then frozen verifier handoff. Preserve Claude a90e3be/two-upstream correction and Codex199a849; no fresh OMP/D2 replay.

Runtime recovery: native Codex CLI subscriber model calls were rejected (no API fallback); scoped native Hermes gpt-6.1-sol subscriber execution passed. A bounded native Hermes continuation will use this existing route, not a new account or daemon. Lease still ends 2026-10-10T05:51:07Z; this receipt does not claim those hours have run.

READY — EL-23-S slice 002 (Hermes)

Head 9b8dd42873102c600c34840c0dbde6c1be55500c; draft PR #35 (nightly, draft=true, autoMergeRequest=null).

Read-only existing-run identity CLI checks exact input/candidate hashes, successful exit, declared count, review-only flags and clean pinned external generator. Real original run-002 artifacts passed; only hashes/counts exported. Producer declarations are not signed execution attestation, freshness, privacy or capability proof.

RED preserved: slice-002-red.log, exit1. Exact-head GREEN: CPU suite 272 tests / 1 skip; 1 producer tests, gym-smoke --n 8, diff-check exit0. No CPU result certifies training/CUDA or generated skill correctness. Mutation n/a. Raw inputs/drafts remain private, uninstalled; source rights/privacy and independent same-cohort qualification remain pending.

Next selected task: Executable refusal controls and receipt pinning, then source-lineage eligibility.

READY — EL-23-S slice 003 (Hermes)

Head 58068d96f26413a29fe78c79c6b3feac9e0b638c; draft PR #35 (nightly, draft=true, autoMergeRequest=null).

Pinned receipt bytes and strict duplicate-key parsing prevent silent rebinding; unaccounted output is refused. Executable controls cover failed-generator stale drafts, missing/changed candidates, changed corpus/ref, dirty checkout, malformed/partial declarations, symlinks and review-only flags. RED observed duplicate success keys and extra files accepted; GREEN refuses both. Real run-002 passed at exact head.

RED preserved: slice-003-red.log, exit1. Exact-head GREEN: CPU suite 284 tests / 1 skip; 13 producer tests, gym-smoke --n 8, diff-check exit0. No CPU result certifies training/CUDA or generated skill correctness. Mutation n/a. Raw inputs/drafts remain private, uninstalled; source rights/privacy and independent same-cohort qualification remain pending.

Next selected task: Source-lineage ambiguity and all-branch prefix exposure eligibility, without changing verifier folds or labels.

READY — EL-23-S slice 004 (Hermes)

Head bd4062b7e769a37cfe40d21204b1be58d4cb6df6; draft PR #35 (nightly, draft=true, autoMergeRequest=null).

Private complete record/session/work-item hash sidecar rejects missing/ambiguous groups and native-session splits. Synthetic two-leaf control counts the shared 2-message prefix as one source group. Real active corpus: 9 records/9 sessions, conservatively quarantined as 1 qualification family, not invented native work-item identity; no repeated leaf prefixes. All groups remain producer-asserted, not independent lineage proof.

RED preserved: slice-004-red.log, exit1. Exact-head GREEN: CPU suite 290 tests / 1 skip; 19 producer tests, gym-smoke --n 8, diff-check exit0. No CPU result certifies training/CUDA or generated skill correctness. Mutation n/a. Raw inputs/drafts remain private, uninstalled; source rights/privacy and independent same-cohort qualification remain pending.

Next selected task: Frozen original-sampler/input identity and paired-experiment protocol to the existing z0evals gate; no candidate execution before privacy/source approval.

READY — EL-23-S slice 005 (Hermes)

Head 6443ea336be0a4f927c9e053cc7734461689fb68; draft PR #35 (nightly, draft=true, autoMergeRequest=null).

Frozen hash-only handoff uses existing z0evals paired scorer at ffa5966, not a new judge. Original sampler replay is deterministic: 1 of 9 records, 12000 characters, truncated/non-complete JSON; reconstructed request hash is not historical wire attestation. Same-model/same-cohort native conditions and tool/error/accounting requirements are explicit; all unmeasured trial fields remain null. Candidate execution BLOCKED on independent privacy/source-rights review, held-out cohort and actual native paired evidence. Historical CI stale-head receipt failure retained.

RED: n/a (metadata/protocol-only slice); behavioral RED receipts for slices002-004 retained. Exact-head GREEN: CPU suite 290 tests / 1 skip; 19 producer tests, gym-smoke --n 8, diff-check exit0. No CPU result certifies training/CUDA or generated skill correctness. Mutation n/a. Raw inputs/drafts remain private, uninstalled; source rights/privacy and independent same-cohort qualification remain pending.

Next selected task: One remaining producer integrity check: bind lineage parsing to the same corpus bytes hashed, then stop if approvals/independent-run dependencies remain blocked.

READY — EL-23-S slice 006 (Hermes)

Head c7009db5f6813ded46ff9e23183f38d5c4d8cd13; draft PR #35 (nightly, draft=true, autoMergeRequest=null).

Lineage inventory now hashes the same complete byte stream it parses, refusing changed blank lines/line endings as well as record drift. Strict parsing rejects NaN/Infinity across receipt, source and sidecar. RED: changed source stream and all three nonstandard constants were incorrectly accepted; GREEN23 producer tests, real frozen artifacts still pass. Immutable handoff at6443ea3 remains unchanged; this is integrity, not native behavior/privacy/efficacy proof.

RED: slice-006-red.log: producer CLI behavioral regression exit 1; earlier generator invalid JSON retained. Exact-head GREEN: CPU suite 294 tests / 1 skip; 23 producer tests, gym-smoke --n 8, diff-check exit0. No CPU result certifies training/CUDA or generated skill correctness. Mutation n/a. Raw inputs/drafts remain private, uninstalled; source rights/privacy and independent same-cohort qualification remain pending.

Next selected task: Refresh independent dependencies and stop as BLOCKED if no safe nonoverlapping producer work remains; release only EL-23-S lease.

COMPLETED_OR_BLOCKED — EL-23-S session closeout / task-specific lease RELEASED

Producer boundary work COMPLETED; actual skill qualification BLOCKED. Final head c7009db5f6813ded46ff9e23183f38d5c4d8cd13, draft PR #35 (nightly; draft=true, autoMergeRequest=null). Five successive continuation slices after9a718a8: 9b8dd42 identity; 58068d9 refusal controls; bd4062b source grouping/prefix accounting; 6443ea3 frozen independent handoff; c7009db same-read integrity.

Exact latest head: CPU294 tests/1skip, 23 producer tests, gym-smoke --n 8 and diff-check exit0; real private original artifacts pass identity/lineage checks. Behavioral RED receipts retained; metadata-only RED/sabotage n/a, mutation n/a. Latest exact-head test/receipt runs passed before closeout body edit; any new receipt remains its actual pending/completed state. Earlier stale-head receipt failure38018805230 remains recorded, not bypassed.

Immutable handoff: producer6443ea3, handoff SHA256 a12002b1422337e22660f2fd00aa143f916c2211597ffc37ce596f45cef2de0c; verifier handoff kvnloo/z0evals#93 (comment). Drafts remain private/uninstalled/unqualified. Original sampler replay confirms one truncated12000-char excerpt; reconstructed request is not captured wire evidence. All9 source sessions are conservatively exposed as one family, not invented native work-item identity.

Blockers: independent source/privacy/rights approval, trusted disjoint cohort and actual same-model/same-cohort native paired behavior/predictions. ctx/native measurement joins remain blocked/unavailable. No gold, efficacy, KEEP, savings, trial usage, installation or adoption claimed. No judge/extractor/runtime writes, model downloads, new workers, paid experiment calls or fallback route.

Observed continuation elapsed 1245.5s from first observed02:47:57Z through2026-10-10T03:08:42.520818+00:00; not a claim that3.5hours ran. No subscriber auth/quota/runtime limit was encountered in this continuation; available quota is unknown. Stop reason is genuine safe-work/dependency exhaustion, before05:51:07Z; no idle/sleep budget filling.

Release ONLY the EL-23-S task-specific lease now; Claude EL-23-X, Codex ZE-93-V and the whole multi-owner issue remain untouched/open. Local consolidated report/session-result are in the owned operation receipts store.

READY — actual native first-proposal comparison, independently verified negative outcome

PR95 head6acc392cb1805c64096e49ffa8a407b544de77e1: exact-head CI38023879183 and latest evidence receipt38023893457 passed. Earlier synchronize receipt38023879175 failed before description head update; remains historical. Source-code scorer pinffa5966; new commit records real evidence only. No default merge.

Native OMP18.8.1, frozen SDK1744f67/native18.8.1, local cached Qwen2.5-Coder7B weight layers, actual32768 context and reviewed first-proposal hash7b3b31f4...; same frozen two-work-item cohort/source revisions/permissions. Native CLI ingested the exact private body through @file; no installation, remote body delivery or mocked decision. Both complete conditions made0 inspection calls and answered0/2 correctly. Production group-macro scorer returned DISCARD, baseline0.0/candidate0.0/delta0.0, query1. Result digest349af6745d37b8e9c1c73578c56a284ec6e49ecdf8b04ceaa7ec9e6d62210215; serialized receipt SHA3a396dc5cadf9d702098bbc4582917071b04315799017e9cd217dae00e5c33fd.

tests_passed=true (owning/producer/evaluator boundaries). agent_behavior_observed=true, agent_behavior_verified=false for successful task behavior. Candidate input verified; successful source inspection/reuse/improvement NOT verified. One proposal even falsely reported execution_completed=true. Presence-only and literal slash-command attempts did not deliver the body and cannot certify efficacy. Native warm/cold state was unmatched, so no latency/savings/repeatability claim; host joules unknown. Protected sealed-score provenance does not override DISCARD. Second write-oriented draft remains unexecuted/unqualified; privacy/redistribution/adoption gates remain.

Queue refresh: producerEL23 lease released atc7009db. Sourcee1c6596 still offline-green. Next independent eligible integration task is actual cold FFF readiness: canonical existing worker returned OBSERVE with FFFException: File system watcher is not ready, despite ready/scanning=false status. No bypass or new resolver; investigate existing file_search implementation under a new bounded#137 claim. The initial memory-scope caller used a component name rather than its canonical repo slug; that configuration error is retained and will be corrected, not solved by relaxing equality.

READY (Claude lane): the usage double-count from my note above is fixed. Supports z0intelligence #144 (the extraction feeds its usage accounting). Repo kvnloo/ai-data-extraction.

Branch factory/claude/claude-code-extractor-subagents-dedupe @ 1d88a046561360cb89aee45b1e80ab64259c7402, two commits on 861f523. Use this branch; it contains the earlier sub-agent work.

A sub-agent transcript found in both directory layouts is now exported once. The other path is recorded in duplicate_source_files. Usage summed one conversation at a time no longer counts it twice.
Two transcripts that share an agent id but differ in content are both kept. Transcripts with no entry ids are never merged, since they cannot be compared.
A padded sessionId now links to its session; a blank one links to nothing.
RED: 2 failures with the new tests on 861f523. GREEN: 69 tests OK. Three mutations of the dedupe code are each caught by a test. An earlier mutation check found one redundant condition, which the second commit removes.

Still open: double counting through a symlinked session directory, and subagent_sessions listing agents whose file was empty. Not independently re-verified. Cost: about 6 tool calls in the main session.

Correction (Claude lane): the dedupe branch I marked READY above had a regression I introduced. Use this branch instead. Supports z0intelligence #144. Repo kvnloo/ai-data-extraction.

What an independent verifier found on 1d88a04

A crash I caused. My second commit removed a guard I called redundant. Without it, a sub-agent entry whose uuid is a list or object aborted the whole export. My mutation check had wrongly shown the guard as removable; the test that would have caught it did not exist.
Copies were merged on entry ids alone. Two copies with the same ids but different content were merged and one was silently discarded. With usage 20 in one copy and 999 in the other, the export reported 30 where the other copy says 1009.
Meta lost on merge. The copy beside the session files always won, so the meta file beside the nested copy was dropped.

Fix: factory/claude/claude-code-extractor-subagents-content-key @ 3ae3faff67bb194ae0e6a7cacb6676cd0e1470f0, one commit on 1d88a04.

Two copies are the same only if their messages are identical. Otherwise both are kept.
A transcript without usable entry ids is never merged, and a non-string id no longer stops the export.
Meta fields from either copy are kept.
RED with the three new tests: 2 failures and 1 error. GREEN: 72 tests OK. Four mutations of the new code are each caught by a test.

Still double counted (the verifier measured these; not fixed)

A copy that is a strict prefix of the other, as when a transcript is copied while still being written: both are exported, the shared turns count twice (130 against a true 100).
The same entries in a different order: fully double counted.
The same transcript under two parent sessions: counted once per session.

For EL-23 and #144: deduplicate on message id before summing usage. This branch removes the exact-copy case only. It has not had its own independent verification.

Cost. Verifier of 1d88a04: 14 tool calls; 63,858 uncached input, 667,258 cache-read, 4,238 output tokens recorded. Fix: about 4 tool calls in the main session, which exposes no per-item token figure.

Independent verification of 3ae3faf found one more way the export could abort; fixed (Claude lane). Use ai-data-extraction factory/claude/claude-code-extractor-subagents-content-key @ 66c27b6b6e2e8d93985d5d10253de4b77bcd2004 (one more commit on 3ae3faf). Supports z0intelligence #144.

What the verifier found on 3ae3faf

The earlier fixtures all pass: a non-string id no longer aborts; same ids with usage 20 versus 999 are both kept; the nested copy's meta survives.
New abort. Hashing the messages can raise on content nested tens of thousands of levels deep, and that call sat outside the per-file guard. On Python 3.14 one such file aborted the whole export; the parent commit exported all three conversations.
Stricter than before in one way. Copies that are equal but spelled differently (20 versus 20.0, or two Unicode normal forms) are no longer merged, so their usage counts twice per conversation. Nothing is dropped, and deduplicating on message id still gives the right total.
When both merged copies have different meta, the first one wins and the other is recorded nowhere.

Fix. A transcript whose content cannot be compared is exported as its own conversation. One new test: RED 1 error, GREEN 73 OK. This last commit has not been independently verified.

Cost: verifier 15 tool calls (68,582 uncached input / 937,081 cache-read / 3,700 output tokens recorded); fix about 2 tool calls in the main session.

Verify, repair, re-verify of the sub-agent export (Claude lane). Improved again; one class of input still aborts. Supports z0intelligence #144. Repo kvnloo/ai-data-extraction.

66c27b6. The verifier ran the whole pipeline through main() with a synthetic home. My guard only moved the failure: content nested about 50,000 levels in a sub-agent file passed the comparison and then aborted when the output was written, leaving a truncated file. A lone surrogate in sub-agent text did the same. The extractor before any sub-agent export (0db773e) completes on both fixtures.

Repair: factory/claude/claude-code-extractor-subagents-r2 @ b661ce9bf2d55e64e7d89acae9f00a70803fcd38 (by a separate agent, on 66c27b6). A sub-agent conversation that cannot be written is skipped with a message, text the output encoding cannot hold is written with escapes, and parent sessions are back in their original order.

Re-verification of b661ce9: still not holding.

A sub-agent file whose name is not valid UTF-8 aborts the export with an empty output file, even if the file is empty. The agent id from the file name goes into the parent's subagent_sessions, and the parent is written unguarded. 0db773e completes on the same fixture. This came in with the first sub-agent commit, not with this repair.
The new "not written" message itself can raise when the path has a non-UTF-8 byte and the terminal encoding is strict, losing every conversation after it.

Use b661ce9 over the earlier heads, with those two limits known. Neither is reachable from ordinary file names.

Cost (uncached input / cache read / output recorded): verify 75,188 / 1,122,461 / 3,963 (19 tool calls). Repair 99,057 / 1,105,518 / 5,250 (14). Re-verify 82,444 / 1,053,534 / 4,942 (18).

Independent verification: holds. Use ai-data-extraction factory/claude/claude-code-extractor-subagents-r3 @ 689261bab549d56dffda6905f29aef29adfd8c4c. Supports z0intelligence #144. Written by one agent, checked by a fresh verifier: no regression against the previous head, none against the extractor before sub-agent export.

File and directory names that are not valid UTF-8 no longer abort the export. Such a name is carried as bytes:// plus its percent-encoded bytes, reversibly, and in the same form on the parent's subagent_sessions and on the sub-agent conversation, so the two still join.
Diagnostic messages are safe for any path and any terminal encoding.
For every fixture on which the pre-sub-agent extractor exits 0, this head exits 0 with complete JSONL, and each parent line is byte-identical apart from the added subagent_sessions key. The verifier ran 5,184 fixtures across two interpreters and several locales.

Still open, and equally so before any sub-agent work

A parent session whose own text holds an unpaired surrogate aborts at the write.
A terminal that cannot encode the banner emoji aborts at the first print.

Known ambiguity: a session id read from transcript content that itself looks like bytes://… is indistinguishable from an escaped file name.

Summary of the sub-agent export as it stands at 689261b: sub-agent transcripts in both layouts are exported and linked to their parent; exact duplicate copies are exported once; malformed, deep or oddly named files cost at most themselves. Still double counted: a copy that is a prefix or a reordering of another. Not exported: structured tool output.

Cost (uncached input / cache read / output recorded): author 314,064 / 2,931,025 / 5,646 (28 tool calls); verifier 246,592 / 3,350,156 / 5,991 (40 tool calls).

Fourth round on the sub-agent export: two-copy cases reconcile; three-copy cases do not yet. No regression. Use ai-data-extraction factory/claude/claude-code-extractor-subagents-r4 @ 38e36c9fa9523cccc0203a4fd94a86f3f49eebdb. Supports z0intelligence #144.

Closed for two copies of one transcript

One copy is a prefix of the other (copied while still being written, or cut mid-line): only the longer is exported; the shorter path and the number of shared leading messages are recorded.
Copies that share a start and then diverge: both kept, each naming the other and the shared count.
Same messages in a different order: exported once.
Same transcript under two parent sessions: exported once per session, each naming the other.
139 tests pass on Python 3.14 and 3.11. Regressions against 689261b: none. Against the extractor before sub-agent export: none.

Not closed, found by the verifier

Three copies. With a full copy, a reordered partial copy and an in-order partial copy, the result depends on directory names: in one layout the reordered copy is exported next to the full one with no field linking them (sum 2,058 against a true 1,236); renamed, the same files reconcile.
A partial copy whose entries carry no session id, or the same transcript under two sessions with different agent ids, gets no linking field.

For anyone summing usage from this export: deduplicate on message id. The linking fields make two-copy cases exact; they are not yet sufficient for three or more.

Cost (uncached input / cache read / output recorded): author 250,913 / 2,557,441 / 6,063 (24 tool calls); verifier 110,182 / 1,016,715 / 5,050 (15 tool calls).

Independent verification: holds. Use ai-data-extraction factory/claude/claude-code-extractor-subagents-r5 @ bb849c5284721f6b677207361cfe875be48d15a7. Supports z0intelligence #144. One author, one separate verifier; no regression against the previous head or the extractor before sub-agent export.

What changed. The pairwise, order-dependent duplicate handling is replaced by one grouping over all sub-agent files of a project, compared by the multiset of message digests.

A copy fully contained in another of the same session and agent (exact, reordered, prefix, reordered prefix) is not exported; its path, shared count and whether order matched are recorded on the copy that contains it.
Copies that overlap without containment are all exported, each listing the other and the exact shared count.
Across different sessions or agent ids nothing is dropped; the conversations are linked with the shared count.
Every exported message carries a dedupe_key. Summing usage over distinct keys is the exact method; pair counts alone cannot give the union when three or more conversations overlap.

Evidence. The author's 1,022 synthetic fixtures reconcile on Python 3.14 and 3.11, under every permutation of directory names for up to four copies. The verifier's own 544 fixtures agree on message sets and all sums. The three-copy case that failed last round now reconciles.

Limits

Files containing a message without a usable entry id are never compared, so an exact pair of such files still counts twice even by key.
Which of two equal-content reorderings is exported depends on path order; sums and message sets do not.
Not yet exported: structured tool output (toolUseResult). That is the next slice.

Cost (uncached input / cache read / output recorded): author 432,823 / 4,695,830 / 6,996 (37 tool calls); verifier 132,794 / 1,559,048 / 18,614 (19 tool calls).

Structured tool results in the extractor: two rounds, not yet accepted. One small defect stands between d94f092 and a clean verdict. Supports z0intelligence #144. Repo kvnloo/ai-data-extraction. Each round: one author, one separate verifier.

What the branch adds (factory/claude/claude-code-extractor-tool-results-r2 @ d94f092980a8e14c181576ba5067f8687e9969cd, two commits on the verified head bb849c5):

Each tool message carries structured (the tool's structured output), source_assistant_id, and for a sub-agent launch subagent_ref with the launch's reported token totals, in the same agent id form as the sub-agent conversation, so the two can be reconciled. Assistant messages keep stop_reason.
Bounds: strings over 2,000 characters and lists over 100 items become a marker with kind, length and hash; image data is never embedded; ids are at most 200 characters or absent; a whole result is capped at 16,000 bytes.
Inputs without these fields export byte-identically to bb849c5 (300 generated sessions, both interpreters). 194 tests pass on Python 3.14 and 3.11.

Round 1 (2af9013): rejected. A source link with a broken Unicode character aborted the export with a truncated file, and two id fields were copied at any length.

Round 2 (d94f092): those four findings are closed; the verifier's own sweep with 5 MB in every position found no failure. Still rejected, for:

Time, on ordinary input. Linking results to their calling message uses a list membership test, so a session with very many assistant entries sharing one tool-call id costs quadratic time: 100,000 such entries take 95.7 s against 0.6 s at bb849c5. Output is identical. Introduced in round 1. The fix is small: a set, or building the table only when something is unlinked.
A file encoding that is not UTF-8. With an ASCII locale, a session whose structured result holds a non-ASCII character stops the export, where bb849c5 completes. bb849c5 fails the same way for such text in fields it already exported, so the exposure is narrow.
Size under the ASCII-escaped fallback. One message can grow by up to 48 KB, above the 20 KB target, because the cap is measured in UTF-8 bytes.

Also changed in round 2, for the owner to accept or not: each sub-agent total in subagent_ref is capped at 1,000 bytes; the full value still appears under structured when the result fits.

Head to use today: bb849c5. Do not use either tool-results branch for export runs until item 1 is fixed and verified.

Cost (uncached input / cache read / output recorded): round 1 author 125,762 / 1,660,839 / 3,741 (22 tool calls), verifier 108,818 / 1,019,772 / 4,578 (17). Round 2 author 121,756 / 1,254,597 / 3,589 (17), verifier 139,969 / 1,943,820 / 12,265 (27).

Independent verification: holds. Use ai-data-extraction factory/claude/claude-code-extractor-tool-results-r3 @ 4fee15a540567dc2f07b84d31f463877eb2d3f06. Supports z0intelligence #144. One author, one separate verifier.

Privacy incident during this round, disclosed first. One diagnostic run by the authoring agent read real session history. It set HOME to a synthetic directory but inherited CLAUDE_CONFIG_DIR from the session, which the extractor honors. That run processed 1,194 real conversations and wrote a 470 MB export into the agent's work directory on the local machine. The agent reports it saw only the printed summary counts, did not open the file, and deleted it in its next command. I checked afterwards: the work directory holds no export file, and no export file left in any extractor work directory from today's rounds mentions the real config directory (checked by file count, not by reading contents). Nothing was uploaded or committed. Every later run in this round used a clean environment. Earlier rounds' agents were given the same instruction that proved insufficient, so I cannot rule out that an earlier run did the same transiently; no trace of one remains. Future extractor runs must clear CLAUDE_CONFIG_DIR as well as set HOME.

The fix. Four lines added, two removed, all in the step that links a tool result to the assistant message that made the call: the lookup table is not built when no result needs it, and at most two callers are kept per call id, which is all the "exactly one" rule needs.

Verifier's evidence

Output is byte-identical to round two for 500 generated sessions with the new fields, and to bb849c5 for 300 without them, through the whole pipeline on Python 3.14 and 3.11.
Time is linear. For one export of entries sharing a single call id: 100,000 entries take 0.5 s (bb849c5: 0.5 s; round two: 103 s); 300,000 take 1.7 s.
197 tests pass. The committed timing test did not fail in 20 runs under load. No regression against bb849c5.

Still open, recorded in round two and not part of this verdict

Under a file encoding that is not UTF-8, a non-ASCII character in a structured result stops the export.
Under the ASCII-escaped fallback one message can grow by up to 48 KB.
An entry id that is a lone surrogate stops the export; this predates the tool-result work.

What the branch now gives a consumer: per tool message, the structured tool output (bounded, with markers for anything large), the id of the calling assistant message, and for a sub-agent launch the launch's reported token totals in the same agent id form as the sub-agent conversation, so reported and summed usage can be reconciled.

Cost of round three (uncached input / cache read / output recorded): author 81,727 / 1,210,546 / 6,854 (25 tool calls); verifier 226,915 / 1,289,364 / 7,256 (25 tool calls).

Extractor: output encoding and escaped-size bounds — independently verified (holds)

Repo kvnloo/ai-data-extraction, branch factory/claude/claude-code-extractor-encoding-r2 @ 6504660b08f64d1b8421f22279a3424915c2ae78. Two commits on the verified 4fee15a (tool-results-r3). Nothing merged, no PR opened, nothing upstream.

What changed

41cba5a: output file, transcripts and sub-agent meta are opened as UTF-8 whatever the locale; size limits of the fields the tool-result work added are measured as escaped JSON (source +16/−14).
6504660: an oversized id is refused on character count before it is serialised (source +3/−1, one test).

Evidence

RED on 4fee15a with the new tests: 37 failures and 3 errors (growth 31,947–47,905 bytes against a 20,000 bound; non-UTF-8 locale exits 1 with 0 lines).
GREEN at 6504660: 204 OK on Python 3.14; 204 OK (3 skipped) on Python 3.11.
Encoding: under LC_ALL=C without UTF-8 mode and under real ISO-8859-1 and CP1252 locales, non-ASCII text in each added field now exports completely (the subagents-r5 head bb849c5 did; 4fee15a exited 1). Output bytes are the same in every environment.
Growth against bb849c5: maximum 18,503 bytes per tool message and 217 per assistant message, in both the UTF-8 and escaped paths; 0 messages over 20,000 in 980 generated worst-case lines (4fee15a: 27 over, maximum 54,714).
Byte-identity through main(), both interpreters: 340 sessions without the new fields equal bb849c5; 340 all-ASCII sessions with them equal 4fee15a.
Mutations: 25 of 25 killed on 41cba5a, 6 of 6 on 6504660.

Failed candidate, kept on the record
The first verifier rejected 41cba5a on time: the id check serialised the whole value on every call. One entry with 3,000 tool results naming a 1 MB id took 4.1 s (0.04 s on 4fee15a); 12,000 results with a 4 MB id took 66 s. 6504660 restores the cheap test first; a second separate verifier measured 0.010 s and 0.040 s, and _id equal to 41cba5a over 166,845 generated values and byte-identical output over 986 conversations.

Behaviour changes for the owner to accept or reject

In a single-byte locale (ISO-8859-1, CP1252), a transcript holding bytes that are not valid UTF-8 is dropped with a printed error. Earlier heads passed the bytes into an output file that was not valid UTF-8. UTF-8 locales already behaved this way.
A structured result made mostly of non-ASCII text becomes a marker at about a third (CJK) to a sixth (emoji) of the size 4fee15a kept. stop_reason over 200 characters of JSON becomes a marker (was 16,000 bytes). Ids over 200 characters once escaped are dropped, including ASCII ids heavy in quotes, backslashes, control characters or DEL.
Constant-factor cost, not growing with input: an id that passes the limit costs about 0.67 µs per check (0.13 µs before); at most 1.18x through main() on 100,000 tool results.

Still open (unchanged, not in scope)

The start-up banner (an emoji) stops every head, including bb849c5, when stdout cannot encode it.
A lone-surrogate entry uuid stops the export (pre-existing).

Privacy: every Python run used env -i with a synthetic HOME; discovered installation paths were printed and confirmed inside the work directory before each main(). No real session history was read. Fixtures deleted.

Cost: author 226,503 uncached input / 1,967,597 cache read / 48,460 output tokens; first verifier 125,907 / 1,469,746 / 16,913; second verifier 16 tool calls (token count not measured). The 6504660 commit was made in the main session.

Extractor: two inputs that stopped the whole export — independently verified (holds)

Repo kvnloo/ai-data-extraction, branch factory/claude/claude-code-extractor-export-stops @ 051f1583f8254b368cb616b7182ede41a35ee2c7, one commit on the verified 6504660. Source +35/−31, tests +148/−10. Nothing merged, no PR, nothing upstream.

What changed

Every message main() prints goes through the existing _say helper, so a stdout that cannot show the banner emoji no longer ends the run.
A session whose line cannot be encoded as UTF-8 (a lone surrogate in an id, timestamp or text) is written through the ASCII-escaped fallback sub-agent conversations already had, instead of ending the run.

Evidence

RED at 6504660: exit 1 with no output file under LC_ALL=C without UTF-8 mode and PYTHONIOENCODING=ascii, latin-1, cp1252. A lone surrogate in 23 of 31 tested entry fields exits 1 with a partial file.
GREEN at 051f158: 209 OK on Python 3.14; 209 OK (3 skipped) on 3.11.
Verifier's own matrix: 17 stdout environments × 6 homes × 2 interpreters all exit 0 (stdout on /dev/full exits 120 with a complete file, as before). 384 lone-surrogate cases all exit 0, with exactly one extra line and the neighbouring conversations byte-identical.
No regression against 6504660: output file and stdout byte-identical for 330 generated ordinary sessions on both interpreters; time on a 100,000-entry session within 3%.
Mutations: all 6 fallback mutants and the 7 emoji-line mutants killed. 16 reverted prints of ASCII-only lines survive; the verifier judged them equivalent.

Behaviour change for the owner to accept or reject

This reverses a rule the earlier head pinned in a test ("a session is never dropped or rewritten quietly", so the export stopped). Such a session is now written whole, with every non-ASCII character in that one conversation as a \u escape; parsing the line gives back the same values. Nothing is printed when this happens.

Corrections to the author's report (from the verifier)

A lone surrogate in requestId or cwd did stop the earlier head; the author said it did not. The fix covers both.
The commit message lists fewer affected fields than the fix covers.

Not run

No real non-UTF-8 locale is installed on the host, so ISO-8859-1 was tested only through PYTHONIOENCODING.
stdout as a terminal, or a pipe closed early, was not tested.

Privacy: env -i with synthetic homes throughout; discovered installation paths checked inside the work directory before every main() run. No real history read. Fixtures deleted.

Cost: author 105,799 uncached input / 1,426,432 cache read / 4,049 output tokens; verifier 105,845 / 922,983 / 10,073.

CLAIM — EL-23 experience-to-skill explicit-use pilot (Hermes). Narrow scope: inspect prior drafts; qualify two or three complete authorized local correction/resolution episodes; invoke the unchanged external generator once on an existing local model; explicitly use one candidate through native Hermes skills in an isolated profile on a read-only source task. No infrastructure, production changes, global installation, branch overwrite, paid fallback, or efficacy claim. Private histories and draft bodies stay local. Independent evaluation belongs to a separate Codex reviewer. Stop on an unmet real prerequisite rather than substitute synthetic or truncated input.

EL-23 experience-to-skill pilot — BLOCKED at input qualification; narrow Hermes claim released. Prior private drafts were inspected and not reused: they remain synthetic/truncated-input evidence. This bounded search did not establish two complete real correction/resolution episodes matching the requested failure. One exact-file probe also reproduced an existing extraction gap: the original Codex extractor at 9176c2d returns no conversation for a modern native session using response_item message/tool records. No extractor code was changed. No history corpus, generated candidate, isolated profile, native skill-use session, or efficacy result is claimed. A clearly hand-authored control and private preflight receipts are saved. Next dependency is exact complete eligible episode provenance through a compatible existing extractor, not more infrastructure. Independent Codex evaluation remains separate; no private histories, draft bodies, or local artifact identifiers are published.

Execution-path check for the skill pilot: the 0/2 with no inspection calls was the model server, not the skill. A working path exists with what is installed. Supports z0intelligence #144. No code changed, no branch, nothing installed. Local inference only. Synthetic workspace and synthetic skill; no private history read.

Why neither arm inspected source (reproduced in a fresh isolated OMP 18.8.9 profile)

qwen2.5-coder:7b under Ollama returns its tool call as plain text. With one simple tool, both /api/chat and /v1/chat/completions return tool_calls: null and the call as JSON in content. OMP therefore executes nothing. This holds with the full 32k context, on GPU, with and without the skill. qwen2.5:3b on the same server returns structured tool_calls, and OMP executes them.
Context is cut to 4,096 by default. OMP's request is about 6,570 tokens with skills and rules off. The server log shows truncating input prompt limit=4096 prompt=6570. OLLAMA_CONTEXT_LENGTH in OMP's environment changes only what OMP believes; the server needs num_ctx (a parameter-only alias).
OMP's default endpoint is the CPU-only server (port 11434: no GPU layers, 175 s to first token). The second local server (port 11435) uses the GPU: 14–24 s per run.
omp -p does not expand /skill:<name> — print mode has no skill dispatch in the installed source. RPC mode does, and wraps the body in the native "User invoked the skill" message.

Working invocation (isolated: env -i, scratch HOME, synthetic checkout as cwd)

# parameter-only alias, no new weights
printf 'FROM qwen2.5:3b\nPARAMETER num_ctx 32768\n' > Modelfile
ollama create <alias> -f Modelfile          # then: omp models refresh
# skill at <cwd>/.omp/skills/<name>/SKILL.md
env -i PATH=... HOME=<scratch> TMPDIR=<scratch> OLLAMA_BASE_URL=http://127.0.0.1:11435 OLLAMA_CONTEXT_LENGTH=32768 \
  omp --mode rpc --model ollama/<alias> --no-session --no-rules --no-extensions --no-lsp --auto-approve --no-title
# stdin: {"id":"1","type":"prompt","message":"/skill:<name> <task>"}   — read events until agent_end

Close stdin for any -p use, or it waits on the pipe.

Evidence the model sees the instruction and can use source tools (same question, same workspace, one run each)

Arm	Skill delivered	Tool calls executed	Result
3B, no skill	–	0	asks for project details
3B, `/skill:` via RPC	yes	2 `grep`	names the correct defining source file; misses the test file
7B coder, `/skill:` via RPC	yes	0	tool call emitted as text

A capture proxy in front of the server recorded the request for the skill arm: it contains the skill body, the invocation header, the canary line, and 11 tool definitions (read, grep, glob, bash, …). The 3B model did not follow the skill's last step (the canary is absent from its answer), and it searched the tests with the wrong pattern. One run per arm is a path check, not a skill-quality result.

Remaining blocker

The 7B coder model cannot act in this harness until its tool calls arrive structured. That needs a different model or serving template; I did not change either. The 3B model is the only installed one that acts, and it is weak.
Pre-flight for any future arm: send one request with one tool and require a structured tool call before scoring anything.

Temporary aliases and the proxy are removed. Cost: main session only, about 35 tool calls, no sub-agents, no paid inference.

Recency-weighted workflow discovery and deterministic controls — 2026-10-19

Preserve the current priority at the top of this issue: one useful history-derived skill, actually used. The broader history study must not block that pilot or restart training/platform work. This is an extension of the existing experiment, not a second generator, memory store or supervisor.

First useful slice

For the already-selected recurring failure, inspect a few complete, authorized episodes and their corrections/outcomes. Describe its trigger, inputs, stable steps, genuinely semantic decisions, human-only tradeoffs and successful stopping condition. Compare the current no-skill/manual/generated controls with a deterministic procedure where the task permits one. Keep original library/author provenance; use the existing ai-data-extraction tools directly, and check current usage rights rather than copying their implementation.

The candidate must be observed in the existing consumer, with actual tool use and an independently checked result on a subsequent task. A generated file, installed skill or matching model answer alone is not benefit.

Wider archaeology, incrementally and privately

Produce a source-coverage manifest: accessible source types, actual earliest/latest records, time zone, observed days/sessions, exclusions, gaps and completeness. Target up to five years where available; do not claim five-year coverage from recent GitHub activity or selected chat excerpts.
Reuse existing episode/outcome identities and source adapters. Deduplicate exports, retries and continuations; distinguish workflow instances from individual tool calls and commits. Treat missing observation as unknown, not zero use.
Rank workflow families with transparent evidence: observed frequency/denominator, recent recurrence, measured time/model/attention cost when available, independent outcome quality, implementation effort and risk. Label estimated quantities; do not multiply arbitrary scores into a purported measured ROI.
Show recent 7/30/90-day patterns separately from long-lived preferences. Evaluate configurable recency decay rather than hard-coding the report's illustrative half-life. Explicit current intent/corrections, source authority and verified outcomes outrank recency; preserve older durable constraints.
Each candidate card identifies deterministic code / agent judgment / human judgment, reuse location, decision contract, confidence, evidence gaps, falsifier, valid region and fallback. Repetition alone is not proof a workflow is safely compilable.
Keep raw conversations, local paths, sensitive identifiers and draft skill bodies private; publish only reviewed, sanitized requirements and aggregate evidence. Do not infer blanket access/training consent, and do not treat hashes as anonymization.

Two distinct ABAB loops

Research: A1 incumbent evidence → B1 proposed extraction → A2 counterevidence/falsification → B2 revised hypothesis. Maintain a claim ledger with sources, assumptions, counterexamples, confidence and next measurement. This is synthesis, not an executed benchmark.

Experiment: compare pinned incumbent and candidate on matched/frozen task families with independent checks. Use complementary orders where carryover matters; record warm/cold state. Changing B2 creates a new candidate version requiring fresh evaluation, not permission to pool adaptive results into the old holdout. See z0evals #93.

Return one KEEP / REVISE / DISCARD pilot decision before expanding to additional families. No rule/skill/model is promoted merely because it repeats history or lowers tokens. Runtime extraction and deoptimization remain z0intelligence #56; human delegation remains #55; #144 there is the consumer interface, not the experiment owner.

Navigation Menu

Repository navigation

eval: tool-calling SLM tournament and Agent0-style self-evolution track #23

CURRENT PRIORITY — one useful history-derived skill, actually used (2026-10-10)

2026-10-23 source-discovery correction (no skill benefit claimed)

Smallest next experiment — existing tools, no new framework

Longer-term research (not prerequisites for the pilot above)

Goal

Candidates

Benchmark the actual control problem

A/B matrix

Agent0-style track

Leakage / promotion

Acceptance

36 remaining items

Recency-weighted workflow discovery and deterministic controls — 2026-10-19

First useful slice

Wider archaeology, incrementally and privately

Two distinct ABAB loops

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

eval: tool-calling SLM tournament and Agent0-style self-evolution track #23

Description

CURRENT PRIORITY — one useful history-derived skill, actually used (2026-10-10)

2026-10-23 source-discovery correction (no skill benefit claimed)

Smallest next experiment — existing tools, no new framework

Longer-term research (not prerequisites for the pilot above)

Goal

Candidates

Benchmark the actual control problem

A/B matrix

Agent0-style track

Leakage / promotion

Acceptance

Activity

kvnloo commented on Sep 21, 2026

kvnloo commented on Oct 5, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

36 remaining items

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 10, 2026

kvnloo commented on Oct 11, 2026

kvnloo commented on Oct 11, 2026

kvnloo commented on Oct 11, 2026

kvnloo commented on Oct 20, 2026

Recency-weighted workflow discovery and deterministic controls — 2026-10-19

First useful slice

Wider archaeology, incrementally and privately

Two distinct ABAB loops

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions