Skip to content

Proofs: regrade stored runs when only tests/ changed #139

Open

Description

@phermosomore

Is your feature request related to a problem? Please describe.

When a check changes after the first proof, the proof runs again from scratch: NOP, Oracle and every negative control, plus any stored agent runs. In an agent-optimization loop on a FreeCAD CAD agent, a run with 8 check changes spent about 226 minutes on proofs (4 to 85 minutes each). Most changes touched only tests/, so the agents' and controls' outputs were still valid; only the grades needed redoing. Harbor already supports this (harbor jobs regrade re-runs the verifier on stored trials). The task-create proof, however, fingerprints tests/ and rejects reused runs, so the skill cannot simply be told to regrade.

Describe the solution you'd like

When only tests/ changed, let a proof regrade stored control and agent jobs instead of re-running them, and record that it did (old and new tests/ hashes, the jobs regraded). Re-run only what the change can affect, for example a new negative control.

Describe alternatives you've considered

Regrading by hand outside the skill, which loses the proof record.

Additional context

Example: NVIDIA/nemoclaw-community#213

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions