Controlled pool tests: A/B real task batches before any pool change lands

Work (Drew, 2026-10-27)

Pool changes proposed by pool-intelligence flags or the steward's
hunches don't land on vibes — they run a controlled test first. This
issue builds the standing mechanism:

A test is a small batch of comparable real tasks (same kind/effort
class, drawn from the live queues) run on the candidate
configuration vs the incumbent, in the same window where possible.
Minimum viable batch: 3 tasks per arm for a directional read, 6 for
a config-change recommendation.
Measured per arm: completion, wall-clock, tokens (context-inclusive
input + output), stalls/nudges, rework within 48h (follow-up fixes
on the same issues).
The test harness is a documented procedure + a results template in
the goal workspace; runs are dispatched through the normal queues
with the route pinned by packet (the packet format already carries
routing intent), so tests cost no special infrastructure.
Verdicts are recorded on the flag's issue and in the interpretive
model doc; a passed test hands the steward a PR-ready change, a
failed one retires the flag with its numbers attached.

Acceptance criteria

Procedure + template exist in the goal workspace.
First controlled test executed end-to-end (two arms, real
tasks, verdict recorded) — candidate chosen from the first
pool-intelligence flag or the Pilot: Workhorse-first vs Frontier-plan split on comparable effort/m tasks (routing evidence) #1191 routing trial's leftovers.
Pilot: Workhorse-first vs Frontier-plan split on comparable effort/m tasks (routing evidence) #1191's trial results are folded in as the mechanism's first
historical dataset rather than re-run.

Child of the pool & routing intelligence epic. Filed by Muse
(operator) 2026-10-27.

CODE-446

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-s
Session / ID: task 5571
Found via: queue claim
Task: task 5571 (small_fix, issue Controlled pool tests: A/B real task batches before any pool change lands #1342 slice 3 — execute the first live controlled pool test, flag issue pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409 vs incumbent) assigned to the opencode execution lane; handing off to the laptop claimant. Prior blockers (quota wall at the 2026-11-02 read; no pinned-route dispatch) are claimant scope to re-check fresh — a stale quota read never idles a pool.
Expectation: claimant launches the bound opencode session; done = first live test executed per issue Controlled pool tests: A/B real task batches before any pool change lands #1342 with the 48h follow-up measurement scheduled.
Links: issue Controlled pool tests: A/B real task batches before any pool change lands #1342, issue pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-y
Session / ID: task 5571
Found via: queue claim
Task: task 5571 (small_fix) on issue Controlled pool tests: A/B real task batches before any pool change lands #1342 — controlled pool tests slice 3: execute the first live test, flag pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409 vs incumbent. Prior progress notes a standing blocker: a controlled A/B result cannot be claimed without a pinned/resolved route — the claimant session should re-check quota/pinned-route blockers fresh.
Expectation: handed off to the opencode execution lane — the laptop claimant launches a bound opencode session for task 5571, which drives it to a terminal state. This lane verified issue Controlled pool tests: A/B real task batches before any pool change lands #1342 OPEN with no hard-stop labels and synced claim labels.
Links: task 5571

🤖 Agent note · handed off

Agent: queue worker (repo-worker-code), lane code-g
Session / ID: task 5571
Found via: queue claim
Task: Task 5571 is assigned to the opencode execution lane; handed off to the opencode claimant, which launches the bound opencode session for the controlled pool test on issue Controlled pool tests: A/B real task batches before any pool change lands #1342 / flag issue pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409.
Expectation: The bound opencode session drives task 5571 to a terminal state under the queue worker id; if it has not launched by lease expiry, task 5571 returns to the queue.
Links: issue Controlled pool tests: A/B real task batches before any pool change lands #1342, flag issue pool flag: add Kimi K3.1 Pro via opencode-go to summit — AA 55, plan door found (models.dev Go catalog + 53/57 measured probe successes), $2.11/task #1409

🤖 Agent note · handed off

Agent: fleet monitor
Session / ID: task 5571 (repo-fleet-monitor run 2026-11-05T15:32:18Z)
Found via: monitor reconciliation (Duty 1 claim/label divergence repair)
Task: Claim evaporated: task 5571 is queued and unclaimed in repo-health-code-pilot, but this issue still carried claimed labels. Labels reset to the truth (queue/queued + status/needs-triage). Checkpoint stage at evaporation: handed_off.
Expectation: The next lane claim picks task 5571 up from the queued state and posts a fresh pickup note.

Navigation Menu

Repository navigation

Controlled pool tests: A/B real task batches before any pool change lands #1342

Work (Drew, 2026-10-27)

Acceptance criteria

111 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Controlled pool tests: A/B real task batches before any pool change lands #1342

Description

Work (Drew, 2026-10-27)

Acceptance criteria

Activity

developing-today commented on Oct 27, 2026

111 remaining items

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 4, 2026

dezren39 commented on Nov 5, 2026

dezren39 commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions