Repository navigation
v083 Arc 2 (WIP): pr_runtime F2P-relax + sweep harness fix - #32
Closed
adithya-s-k wants to merge 3 commits into
Closed
adithya-s-k wants to merge 3 commits into
adithya-s-k wants to merge 3 commits into
Conversation
Plan §4 Arc 2 known issue: ~17-27% of candidates are dropped on 'no_fail_to_pass' because the PR's test_patch added or modified tests that already passed before the fix. Currently we skip; these candidates are real bugs in production code (the PR landed for a reason) but the SWE-bench-style F2P bar is too strict for them. Add `PRRuntimeOptions.allow_no_f2p_with_test_patch` (default False — preserves current strict behavior). When set True via --pipeline-opt: accept the candidate IF F2P=0 AND test_patch is non-empty AND there's at least one passing test. The verifier still gates on post-patch behavior, so a no-op model patch would have to pass the modified tests too. The F2P decision logic is extracted into a pure helper `_should_skip_no_f2p` and unit-tested with 6 new cases (default strict, relaxation accept/skip flavors, F2P-meets-bar, require-off short-circuit). Full suite at 655 passing.
Two harness fixes surfaced by the Arc 2 smoke against pallets/click: 1. `repo2rlenv generate` exits 1 when emitted=0 (cli.py:275). For pr_runtime that's a normal outcome — many PRs are correctly filtered on `no_test_patch` or `no_fail_to_pass`. The sweep was misreading that as a cell failure. Fix: when subprocess exits non-zero AND the out_dir exists AND no tasks landed, treat as `done` with `candidates=0`. True failures (out_dir missing entirely) still land as `step=failed`. 2. Wire `--pipeline-opt KEY=VALUE` end-to-end so arc-specific options like `allow_no_f2p_with_test_patch=true` can be set without editing the script. Forwarded to `repo2rlenv generate` for each cell. 3 new unit tests for the option parser (basic shapes, empty, rejects bare keys). Suite at 658 passing.
Documents the v0.8.3 Arc 2 work-in-progress: - F2P filter relaxation as an opt-in flag (with rationale + 6 tests) - Sweep harness fix discovered while smoking: don't fail cells when generate exits 1 with 0 tasks emitted (which is a normal pr_runtime filter outcome) - --pipeline-opt plumbing through sweep.py - Honest record of the limited smoke evidence (10 PRs across click+werkzeug, none matched the relaxation pattern; full sweep deferred pending cost approval)
This was referenced May 20, 2026
Collaborator
Author
|
Closing as part of v0.8.3 cleanup — restructuring to one-PR-at-a-time per request. Work preserved on the branch; see PR #31 for the active cleaned-up arc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arc 2 — pr_runtime (work in progress)
Per plan §4 Arc 2. Stacked on #31 (Arc 1 pr_diff) which is stacked on #30 (sweep harness).
This PR lands the optimization + tests; the full 38-repo sweep is deferred until cost approval. See the findings doc for full context.
Summary
Optimization — opt-in F2P filter relaxation
Plan §4 Arc 2 known issue: ~17-27% of candidates drop on
no_fail_to_passbecause the PR's test patch added or modified tests that already passed before the fix landed. These are real production bug fixes whose test coverage doesn't follow the strict SWE-bench shape.PRRuntimeOptions.allow_no_f2p_with_test_patch: bool = False.When set via
--pipeline-opt allow_no_f2p_with_test_patch=true, the pipeline accepts F2P=0 candidates iff the test_patch is non-empty AND the P2P set has ≥1 entry._should_skip_no_f2pfor testability.tests/test_pipeline_pr_runtime.pycovering the strict path + each relaxation guard.Harness bugfix — discovered by the smoke
Smoke surfaced that `repo2rlenv generate` exits 1 when emitted=0 (cli.py:275). For pr_runtime that's a normal filter outcome — the sweep was misreading it as a cell failure.
scripts/v083/sweep.pynow treats "exit non-zero + out_dir exists + no task dirs" asstep=done, candidates=0. True crashes (out_dir missing) still land asfailed.Smoke evidence (limited)
step=failedbecause of the exit-code issue; pipeline itself worked. Surfaced the harness bug.no_test_patch, 1no_fail_to_pass(P2P=0 → relaxation correctly didn't rescue)The 10 most-recent PRs in click+werkzeug happen to not include any matching the relaxation pattern. To exercise the rescue path requires historical PRs (
--pipeline-opt since=2024-01-01) or higher-activity repos likepydantic/pydanticorpsf/requests— deferred to the full sweep.What's pending for full Arc 2 completion
AdithyaSK/repo2rlenv-v083-pr_runtime.Test plan
uv run pytest -q— 658 passing, 2 skipped (+6 new F2P-gate tests, +3 new option-parser tests)uv run ruff check .+uv run ruff format --check .— cleansweep.py --pipeline pr_runtime --repos pallets/click pallets/werkzeug --pipeline-opt allow_no_f2p_with_test_patch=true→ 2/2 cellsstep=doneharbor run --agent oraclesmoke on a random task — pending sweepOut of scope
Notes for reviewer
pyproject.tomlbump (stays at0.8.2.post3per plan §5).Co-Authored-Bytrailer.