pr_runtime pipeline (v0.3): SWE-bench-style PR mining with sandbox-verified oracles #3

Summary

Implement the pr_runtime pipeline — the first sandbox-verified pipeline, the natural evolution of v0.1's pr_diff. For each merged PR we run the test suite twice inside the bootstrap container (once with test_patch alone to capture which tests fail pre-fix, once with patch + test_patch to confirm which now pass), then emit a Harbor task whose oracle is the FAIL_TO_PASS / PASS_TO_PASS sets and whose reward is test_execution (binary).

This unlocks the first verifiable-via-execution dataset format the project ships.

Design doc

See docs/pipelines/pr_runtime.md — full algorithm, eval-script shape, lite-filter heuristics, what we reuse from references/SWE-bench/.

Internal scaffolding: plans/pr_runtime_plan.md — 6 phases + risks + open questions.

Inspiration

Mirrors SWE-bench's task shape, validation flow, and grading semantics:

• SWE-bench (Princeton, ICLR'24, arXiv:2310.06770) — the original PR-as-task formulation; we adapt their collect/utils.py:extract_patches (diff split), harness/test_spec/utils.py:make_eval_script_list_common (eval script), and harness/grading.py (FULL/PARTIAL/NO resolution).
• SWE-bench-Live (Microsoft, NeurIPS'25, arXiv:2505.23419) — productionization recipe (automated env-build via RepoLaunch, polyglot extension). We already shipped the equivalent of RepoLaunch as our bootstrap/ in v0.2.

We study these reference implementations and don't vendor any code — independent Apache-2.0 implementation with acknowledgment blocks.

Approach

Two architectural calls:

1. Auto-trigger bootstrap from generate. Pipelines declare requires_bootstrap: ClassVar[bool] = True. cmd_generate inspects this before constructing the pipeline; on True it calls ensure_bootstrap() with the live BootstrapView UI. Cache hit ⇒ instant. The pipeline __init__ gains an optional bootstrap: BootstrapResult | None arg. Lite pipelines (pr_diff) keep requires_bootstrap=False and ignore the arg.

2. Shared sandbox + git reset --hard between PRs. Unlike SWE-bench (which builds one image per instance), we share one bootstrap image across PRs and reset the working tree between each. Cheaper on disk + faster. Each validation stage script starts with git reset --hard <base_commit> + git clean -fdx for isolation.

Architecture

GenerationInput
       │
       ▼
cmd_generate (cli.py)
  ├─ if pipeline.requires_bootstrap:
  │     ensure_bootstrap() ── BootstrapView ── cached under envs/
  │
  └─ pipeline(input, options, bootstrap=BootstrapResult)
        ├─ list_merged_prs()                  via gh CLI
        ├─ split_patch_and_test_patch()       SWE-bench keyword-path heuristic
        ├─ validate_pr() inside DockerSandbox
        │     ├─ stage 1: reset → apply test_patch → pytest -v → pre_status
        │     └─ stage 2: reset → apply patch+test_patch → pytest -v → post_status
        │           └─ F2P = (pre=FAIL ∧ post=PASS)
        │              P2P = (pre=PASS ∧ post=PASS)
        └─ write_harbor_task()                + environment/Dockerfile + tests/test.sh

What ships

What you can do once this lands

# Auto-bootstraps (cached after first run), mines + validates PRs, emits Harbor tasks
repo2rlenv generate \
  --repo <owner>/<repo> \
  --pipeline pr_runtime \
  --pipeline-opt limit=20 \
  --llm anthropic/claude-sonnet-4-6 \
  --out hf://<your-org>/<dataset>

# Each task is verifiable end-to-end:
harbor run -d ./datasets/<dataset>/<task> -a oracle -e local-docker  # reward 1.0

Out of scope (deferred to v0.4)

• Polyglot log parsers (JS/Go/Rust). Python only in v0.3.
• Flaky-test retry / majority voting.
• Parallel per-PR validation.
• LLM-judged QA gate (SWE-Bench++ four-layer recipe).

Success criteria

• All 71 v0.2 tests still pass + new unit tests for diff split, pytest parser, grading, eval-script construction
• Pipeline contract test confirms pr_runtime conforms to Protocol with requires_bootstrap=True
• End-to-end: generate emits valid Harbor tasks against a real repo (verified on pallets/click)

Tracking PR: feat/pr-runtime-v0.3.