Define an evidence-first contributor workflow and retain rejected semantic experiments

[sims1253]

Motivation

The default unit of human or AI-assisted work should be a falsifiable hypothesis with retained evidence, not a plausible patch plus an aggregate green result. Rejected approaches should stay discoverable so later contributors do not unknowingly repeat them.

Foundation and motivating examples

CONTRIBUTING.md already sets a strong false-positive bar, ordered gates, and R-backed semantic claims. This proposal operationalizes those requirements; it does not replace them with generic agent instructions.

Two recorded examples motivate it:

• Model classed callback dispatch without losing data-frame diagnostics #324: a broad class-dispatch guard addressed genuine dispatch mistakes and left the 500-package ecosystem comparison unchanged, yet removed a valid data-frame arithmetic diagnostic (RY040). Only the authored true-positive control caught it.
• Track the remaining enforced R-oracle known gaps #323: a broader branch-refinement experiment was rejected because unknown calls and superassignment invalidated its assumptions; that boundary must be respected, not the rejected patch restored.

These are recorded historical findings, not experiments rerun for this proposal.

Proposed first increment

Add a short contributor playbook and an applicable PR/experiment template for semantic, performance, and concurrency changes. Keep it optional and lightweight for documentation-only changes. Link from CONTRIBUTING.md rather than introducing a separate, conflicting authority.

The workflow should cover:

• Hypothesis and boundaries: the mechanism, expected observable result, preserved true positives, quiet neighboring idioms, and the evidence that would falsify the proposal.
• Small discriminating experiment: establish the baseline witness and R semantics where relevant before widening a fix; optimize one mechanism at a time; do not select a patch by headline totals.
• Layered validation: targeted controls first, then the unchanged required gates and relevant real-source/performance checks; distinguish a local speedup from a demonstrated end-to-end benefit.
• Independent verification: a reviewer or separate verification run reproduces critical claims from source identities, commands, and raw outputs. An agent's narrative or another model's agreement is not a substitute for an oracle.
• Durable outcome: retain the accepted invariant or rejected mechanism, smallest counterexample, revision identities, evidence location, and a brief reason. Link existing issues/PRs instead of copying entire transcripts.

For AI-assisted work, the same contract applies regardless of provider. Agents may propose experiments and patches inside explicit scope, but semantic contract changes, baseline/ledger acceptance, and merging stay subject to review. No extra repository write permissions or automatic agent workflow is implied.

Acceptance criteria

• A concise template plus one worked example based on Model classed callback dispatch without losing data-frame diagnostics #324, clearly identified as a documented historical example rather than a fresh benchmark: the example rejects a candidate that preserves aggregate corpus counts but loses the named true-positive control.
• A small rejected-experiment record defined (hypothesis, invalid assumption, counterexample, relevant revisions, evidence link), with guidance on when revisiting is justified.
• Findings, hypotheses, measurements, unexecuted checks, and unresolved uncertainty are distinguishable; missing tools or evidence cannot be summarized as green validation.
• Changes to oracle premises, known-gap tags, triage labels, benchmark workloads, and reference baselines require explicit review; updating an expectation is not itself evidence of correctness.
• Pilot the template on one bounded change and check whether another contributor can reproduce its key claim without access to the original chat session.

Boundaries

No long research document required for every PR, no mandated model, no auto-generated repository policy, and a rejected patch is not permanently forbidden. This documentation task ships independently of the experiment-report utility proposed in #545: the playbook defines the evidence contract; the utility only reduces the work of collecting it.

[pullfrog]

  ^{｜ Build this ➔ ｜ Make a plan ➔}