Skip to content

Tester-feedback triage: converged statuses, owner decisions, and the batch slate (revisions + next pass + measurement) #24

Closed
Closed
Tester-feedback triage: converged statuses, owner decisions, and the batch slate (revisions + next pass + measurement)#24

Description

@Carr1005
Carr1005
opened on Sep 8, 2026

Triage of the pre-F12 tester feedback (4 new testers + the P1/P2/P3 list), verified item-by-item against the post-F12 state (the PR #23 branch), with the owner's design decisions and the routing we converged on. This issue is the candidate slate for the next batch; nothing here is applied yet.

Verified statuses

# Item Status Evidence / disposition
1 Time/cost/dependency-size expectations Not in spec — owner decided it stays out No wall-clock/cost/GB figure anywhere (embedding model named twice with no size; Oracle cost qualitative only). Owner: cost figures are perishable, machine-dependent data — the spec-philosophy class we deliberately exclude. Instead: the recommended baseline IS the minimum-resource path, and the scaled test measures it (see Measurement below).
2 Footprint/runtime stop rule for the build agent Dropped by owner decision No bound exists in §6/§7; none will be added. The baseline path is the structural resource bound. Note for fairness to the 5.6 GB report: most of that footprint is legitimate dependencies (torch + model), not runaway output. Replaced by heavy-option flags (next-pass item B2).
3 Keyless path not framed as a legitimate finish Mechanism YES, framing NO The 19-offline/6-live split, RUN_LIVE=1 gating, and skip-with-reason reporting all exist and work (run-09 and gate-probe-2 both completed keyless). But no sentence blesses it — "5 skipped" reads as unfinished debt. Fix: the baseline-completion sentence — "the recommended baseline build is complete when the offline suite is green; live ACs are an optional keyed extension; skipping them without a key is the expected result." Also resolves item 7. → batch item A1.
4 Stale model names / OpenAI pinning / parameter constraints Defaults unchanged; scaffolding exists; F12 built the right slot D4's invariant is already provider-agnostic; CTX-D declares gpt-5 names perishable; R4 falls back to 128k for unknown models. Missing: current-model Options examples, a CTX-D "perishable parameter constraints" category (the temperature rejection), §2 key wording. Provenance constraint: "(course default)" stays on the gpt-5 family — provider-neutrality must not overwrite provenance. Current-model advice belongs in the gate-time "(Recommended)" slot F12 defined, not baked into defaults. → batch items A3 (spec) + B3 (guide rule).
5 Seed/fixture ingestion idempotency (P3: verify-only) Split Toolbox re-registration dedup is specified + AC-tested (R9/AC14); store init is create-if-missing/no-wipe (R1/AC1/AC2, restart-tested). Re-running seed ingestion into a populated store is undefined — can't crash in the current realization (generated ids), but silently duplicates. Proposed (not applied): idempotent-seed AC or an explicit "re-seeding duplicates; reset() first" note. → watchlist C3.
6 arXiv 429 (P3: verify-only) Not in text; evidence now n=2 run-09's AC24 hit 429 twice, passed later — transient; the arxiv client retries internally; inside the agent loop a 429 is already non-fatal via R11 (failed tool-log row, turn continues). Proposed (not applied): document transient-429 as expected for live (net) ACs rather than mandate backoff. → watchlist C4.
7 Skipped ACs read as failure (P3: verify-only) Mechanism YES, healthy-shape sentence NO §7's skip rule is right; nothing states what a healthy run looks like per mode. Folded into the baseline-completion sentence (item 3). → batch item A1.
8a File density / no on-ramp (R16, R18 verbatim: "overwhelming at first due to its density"; "needed a better explanations and instructions in the beginning"; R16's ask: "a brief, high-level overview at the top summarizing what the system actually does") NOT addressed First ~100 lines are meta-blockquote → gate instructions → 12-row Ledger; §1 Objective (the first "what does this build" sentence) is buried after all of it. Fix: an orientation block at the top — what you're building + the baseline-completion sentence (no cost numbers, per item 1). Sits with the opening blockquote, before §0, so the gate's "first section" rule is untouched. → batch items A1 (spec) + B1 (guide template).
8b Process ceremony (R17 verbatim: "the process wrapper (how to ask) was heavier than it needed to be"; "I'd lean on the default → checklist → build flow") Addressed by PR #23 R17's requested flow is literally the F12 express lane: one baseline question skips all per-row questions → deviation-marked checklist → build. Gate-probe-2 demonstrated the exact flow. Note: R16 praised the stop-and-ask constraint and R17 called the decision content "excellent" — the objection was front-loading, not the mechanism; no further weakening. → watchlist C1 (confirm with next testers).

Owner decisions locked in this triage

  1. No cost/time/size figures in specs — perishable data; the scaled test measures the baseline instead.
  2. No build-agent stop rule — the recommended baseline is the structural minimum-resource path.
  3. Heavy options get their setup weight inline in the Options-cell text (e.g. "(course default — heavy setup: container + admin credentials, multi-GB image)"), NOT a new (warning:) label — F12's two-sanctioned-labels rule stays intact.
  4. The keyless offline run is the baseline's completion criterion (the blessing sentence).

The batch (executed together after the full issue sweep)

A. Spec revisions batch (Loop A, agent-memory spec.md):

  • A1. Orientation block at top: plain-language overview + baseline-completion sentence (covers 3, 7, 8a).
  • A2. Summary-thread-scoping Ledger row (owner decision recorded in Guide residuals from guide-pins-v2: route closure, constants provenance, fixture churn, gate-semantics reform #18 / the F12 matrix; contradicted route, thread-scoped default, global pool alternative with the isolation-vs-recall trade-off).
  • A3. D4 Options broadened with current-model examples + CTX-D "perishable parameter constraints" category (temperature rejection) + §2 key wording; provenance labels untouched.
  • A4. Heavy-option setup-weight notes inline in D11/D12 (and any other heavy row) Options cells.

B. Next promotion pass (guide; one Opus validation round covers all):

  • B1. Orientation-block template (course-agnostic) in the spec anatomy.
  • B2. Rule: a heavy option states its setup weight inline in its Options entry.
  • B3. Rule: model-currency advice is gate-time "(Recommended)" business, never baked into defaults; CTX-D gains the perishable-parameter-constraints category.
  • B4. From Guide residuals from guide-pins-v2: route closure, constants provenance, fixture churn, gate-semantics reform #18's existing queue: identifier-format mining incl. alphabet (cross-model pinning gap); importable fixture module filenames; summary-scoping example sentence + P1→13 canon update.

C. Scaled-test measurement watchlist (measure, don't write):

  • C1. Post-F12 ceremony weight (R17's complaint — is one baseline question light enough?).
  • C2. Recommended-baseline resource benchmark (wall-clock, tokens, disk) — the numbers item 1 wanted, measured not promised.
  • C3. Ingestion idempotency recurrence (item 5).
  • C4. arXiv 429 recurrence (item 6).

Remaining from earlier, for the full sweep

🤖 Generated with Claude Code

Activity

  1. Carr1005 commented on Sep 9, 2026

    @Carr1005
    Carr1005
    on Sep 9, 2026
    CollaboratorAuthor

    Batch delivered via PR #25:

    A-items (spec): A1 orientation block (overview scene + baseline-completion sentence, honesty-tightened: keys unlock live tests and live app use), A2 the D13 summary-scope row (first contradicted-category row), A3 D4 providers-not-model-names + CTX-D request-parameter perishables + §2 key wording, A4 inline heavy-setup notes (D11/D12). Plus the Section-2 pulls: fixture filename made importable; id-alphabet handled guide-side.

    B-items (guide, all validated 3/3 on Opus-class regens): orientation template + reader test; heavy-option inline-weight rule; model-currency timing rule; id-alphabet mining rule (closed the 0/3 cross-model gap — 3/3 after); importable-filename rule; the scoping-contradiction worked example (scope row held 3/3; one regen produced the first exact-canon 13/13 Ledger).

    Live-AC item — closed, with a find: first full keyed run. AC4/AC9/AC19/AC22/AC24 passed live; AC21 (the continuity demo) failed deterministically — the model is never told its thread id, guessed current_thread, and the summarize tool honestly consolidated nothing. Spec gained the R11 harness-known-identity constraint (living-document rule), the build was conformed, and AC21 passed on rerun: live suite 6/6. The guide generalization (harness-known identities are injected, never model-guessed) is the next pass's first candidate.

    C-watchlist: unchanged (ceremony weight, baseline benchmark, ingestion idempotency, arXiv 429) — all await the scaled test.

    🤖 Generated with Claude Code

  2. Carr1005 commented on Sep 10, 2026

    @Carr1005
    Carr1005
    on Sep 10, 2026
    CollaboratorAuthor

    Closing: every editable item from the tester feedback is delivered (PR #25, merged as e05a825). What remains from this triage is measurement, which needs a tester round rather than an edit — moved to a dedicated issue.

    A-items (spec) — delivered

    Item Delivered as
    A1 orientation block + baseline-completion sentence (fixes the density complaint, blesses the keyless finish, and stops skips reading as failure) Opening blockquote: "what you're building" as a user-visible scene + "complete when the offline suite passes; a key only unlocks the live tests and live use"
    A2 D13 summary-memory-scope row First contradicted-category row: thread-scoped default, unscoped global pool as the course-demonstrated alternative, isolation-vs-recall trade-off
    A3 model currency D4 names providers/protocols not current model names; CTX-D gains request-parameter compatibility (the temperature rejection); §2 key wording
    A4 heavy-option setup weight Inline in the Options entries the gate actually shows (Oracle: container + admin credentials + multi-GB image; Tavily: paid key)

    B-items (guide) — delivered and validated 3/3 on clean-room Opus regens: orientation-block template with the reader test, inline heavy-option weights, model-currency timing rule, identifier-alphabet mining, importable fixture filenames, and the scoping-contradiction worked example.

    Live-AC item — closed with a find. First full keyed run: AC4/AC9/AC19/AC22/AC24 passed live; AC21 (the cross-session continuity demo) failed deterministically because the model was never told its thread id and guessed one. The spec gained the missing constraint per its own living-document rule, the build conformed, and AC21 passed on rerun — live suite 6/6.

    Two follow-up fixes from a disposable-clone trial (the spec copied alone into an empty folder, built by a fresh session; build green, independent eval 9/10) also landed in #25: the R11 rule generalized beyond one named tool, and the gate checklist written to resolved-decisions.md. On review, two of my justifications for the second were overstated and are corrected in docs/research/guide-revisions-batch-matrix.md: the file does not prove the gate ran (it is forgeable — only probes test gate behavior), and the "3/3 builds wrote it unprompted" evidence was inflated. The purpose that survives is a recoverable decision record in a fixed place, and the wording was trimmed to match (guide cost cut from +256 to +104 words).

    Owner decisions from this triage, preserved for future reference: no cost/time/size figures in specs (perishable — the baseline is the cheap path, and we measure it instead); no build-agent stop rule (the baseline path is the structural bound); heavy options disclose setup weight inline rather than via a new label; keyless offline completion is the baseline's completion criterion.

    Moved to the new measurement issue: ceremony weight post-F12, the baseline resource benchmark, ingestion idempotency, arXiv 429 recurrence, whether the orientation block actually cures the density complaint, and whether builds in fact write the checklist file.

  3. Carr1005
    closed this as completedon Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions