Skip to content

v083 Arc 2 (WIP): pr_runtime F2P-relax + sweep harness fix - #32

Closed
adithya-s-k wants to merge 3 commits into
v083-arc1-pr_difffrom
v083-arc2-pr_runtime
Closed

adithya-s-k wants to merge 3 commits into
v083-arc1-pr_difffrom
v083-arc2-pr_runtime

Conversation

@adithya-s-k

Copy link
Copy Markdown
Collaborator

Arc 2 — pr_runtime (work in progress)

Per plan §4 Arc 2. Stacked on #31 (Arc 1 pr_diff) which is stacked on #30 (sweep harness).

This PR lands the optimization + tests; the full 38-repo sweep is deferred until cost approval. See the findings doc for full context.

Summary

Optimization — opt-in F2P filter relaxation

Plan §4 Arc 2 known issue: ~17-27% of candidates drop on no_fail_to_pass because the PR's test patch added or modified tests that already passed before the fix landed. These are real production bug fixes whose test coverage doesn't follow the strict SWE-bench shape.

  • Add PRRuntimeOptions.allow_no_f2p_with_test_patch: bool = False.
    When set via --pipeline-opt allow_no_f2p_with_test_patch=true, the pipeline accepts F2P=0 candidates iff the test_patch is non-empty AND the P2P set has ≥1 entry.
  • Decision lives in a pure helper _should_skip_no_f2p for testability.
  • 6 new unit tests in tests/test_pipeline_pr_runtime.py covering the strict path + each relaxation guard.

Harness bugfix — discovered by the smoke

Smoke surfaced that `repo2rlenv generate` exits 1 when emitted=0 (cli.py:275). For pr_runtime that's a normal filter outcome — the sweep was misreading it as a cell failure.

  • scripts/v083/sweep.py now treats "exit non-zero + out_dir exists + no task dirs" as step=done, candidates=0. True crashes (out_dir missing) still land as failed.
  • Also wires `--pipeline-opt KEY=VALUE` through the sweep CLI so arc-specific options flow without code edits.
  • 3 new unit tests for the option parser; suite at 658 passing.

Smoke evidence (limited)

Sweep Repos envs/cell Outcome
#1 pallets/click 2 sweep reported step=failed because of the exit-code issue; pipeline itself worked. Surfaced the harness bug.
#2 pallets/click 3 3 PRs: 2 no_test_patch, 1 no_fail_to_pass (P2P=0 → relaxation correctly didn't rescue)
#3 pallets/click + pallets/werkzeug 5 each 2/2 cells step=done, 0 emitted, $0 cost. Harness fix verified.

The 10 most-recent PRs in click+werkzeug happen to not include any matching the relaxation pattern. To exercise the rescue path requires historical PRs (--pipeline-opt since=2024-01-01) or higher-activity repos like pydantic/pydantic or psf/requests — deferred to the full sweep.

What's pending for full Arc 2 completion

  1. Full 38-repo sweep — should yield ~75-85 verified envs. Cost envelope: ~$80-250 (bootstrap LLM + T2 Haiku + optional T4 Sonnet). Wall time: 6-12 h.
  2. HF push to AdithyaSK/repo2rlenv-v083-pr_runtime.
  3. Findings update with concrete T3 yield + relaxation-impact delta.
  4. Per-language log-parser polish for Tier C — deferred until we see real Tier C failure modes from the full sweep.

Test plan

  • uv run pytest -q — 658 passing, 2 skipped (+6 new F2P-gate tests, +3 new option-parser tests)
  • uv run ruff check . + uv run ruff format --check . — clean
  • Smoke: sweep.py --pipeline pr_runtime --repos pallets/click pallets/werkzeug --pipeline-opt allow_no_f2p_with_test_patch=true → 2/2 cells step=done
  • Full 38-repo sweep — pending user OK on cost
  • HF push — pending sweep
  • Consumer-side harbor run --agent oracle smoke on a random task — pending sweep

Out of scope

Notes for reviewer

Plan §4 Arc 2 known issue: ~17-27% of candidates are dropped on
'no_fail_to_pass' because the PR's test_patch added or modified tests
that already passed before the fix. Currently we skip; these candidates
are real bugs in production code (the PR landed for a reason) but the
SWE-bench-style F2P bar is too strict for them.

Add `PRRuntimeOptions.allow_no_f2p_with_test_patch` (default False —
preserves current strict behavior). When set True via --pipeline-opt:
accept the candidate IF F2P=0 AND test_patch is non-empty AND there's
at least one passing test. The verifier still gates on post-patch
behavior, so a no-op model patch would have to pass the modified
tests too.

The F2P decision logic is extracted into a pure helper
`_should_skip_no_f2p` and unit-tested with 6 new cases (default strict,
relaxation accept/skip flavors, F2P-meets-bar, require-off short-circuit).
Full suite at 655 passing.
Two harness fixes surfaced by the Arc 2 smoke against pallets/click:

1. `repo2rlenv generate` exits 1 when emitted=0 (cli.py:275). For
   pr_runtime that's a normal outcome — many PRs are correctly filtered
   on `no_test_patch` or `no_fail_to_pass`. The sweep was misreading
   that as a cell failure. Fix: when subprocess exits non-zero AND
   the out_dir exists AND no tasks landed, treat as `done` with
   `candidates=0`. True failures (out_dir missing entirely) still
   land as `step=failed`.

2. Wire `--pipeline-opt KEY=VALUE` end-to-end so arc-specific options
   like `allow_no_f2p_with_test_patch=true` can be set without editing
   the script. Forwarded to `repo2rlenv generate` for each cell.

3 new unit tests for the option parser (basic shapes, empty, rejects
bare keys). Suite at 658 passing.
Documents the v0.8.3 Arc 2 work-in-progress:
- F2P filter relaxation as an opt-in flag (with rationale + 6 tests)
- Sweep harness fix discovered while smoking: don't fail cells when
  generate exits 1 with 0 tasks emitted (which is a normal pr_runtime
  filter outcome)
- --pipeline-opt plumbing through sweep.py
- Honest record of the limited smoke evidence (10 PRs across
  click+werkzeug, none matched the relaxation pattern; full sweep
  deferred pending cost approval)
@adithya-s-k

Copy link
Copy Markdown
Collaborator Author

Closing as part of v0.8.3 cleanup — restructuring to one-PR-at-a-time per request. Work preserved on the branch; see PR #31 for the active cleaned-up arc.

@adithya-s-k
adithya-s-k deleted the v083-arc2-pr_runtime branch May 21, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant