opencode-eval-runner

Reusable OCI execution boundary for isolated OpenCode or GitHub Copilot CLI invocations, plus evaluation orchestration over project-supplied profiles or explicit runner-owned input families.

The runner does not own a project eval corpus or agent policy. In --profile mode, the project profile also owns grading semantics; built-in modes implement only the accepted agent/runtime and skill-family semantics. This repository owns two boundaries:

• invoke: exactly one fresh isolated model invocation;
• eval: generic selection, iterations, concurrency, attempts, evidence readiness, target/judge orchestration, classification, artifacts, summaries, and exit status over either a project-supplied EvalProfile or the built-in agent/runtime and skill input families.

Each actual model invocation still gets a fresh container, fresh HOME/XDG/OpenCode state, explicit provider configuration/authentication, and a read-only workspace by default.

Public interfaces

opencode-eval-runner invoke ...
opencode-eval-runner eval ...

invoke remains the one-invocation primitive and has no hidden retry loop.

eval preserves the external --profile module:attribute mode and adds built-in modes for explicit agent/runtime or skill-case inputs. Profile mode leaves case and grading semantics with the profile. Built-in mode uses the runner's accepted family contracts; its live target and judge invocations are OpenCode-only. Combined skill runs accept repeated explicit input/skill bindings; they do not scan directories. See the Public eval CLI and profile reference for input selection, output formats, compatibility, and execution safeguards.

Supported usage modes include:

Mode	Interface
Local OpenCode invocation	opencode-eval-runner invoke --transport opencode ...
Local Copilot invocation	opencode-eval-runner invoke --transport github-copilot-cli ...
Local generic eval	opencode-eval-runner eval --profile module:attribute ...
Local built-in eval	opencode-eval-runner eval --input cases.json --family agent-runtime or explicit skill-family inputs
GitHub Action direct invocation	Action inputs with model set
GitHub Action generic/project eval	Action command mode
GitHub Action setup only	Leave model and command empty, then call the CLI later

See Public eval CLI and profile reference for selection safeguards, profile loading, iterations/concurrency, target/judge overrides, retries, artifacts, and exit codes.

See Invocation usage and interface reference for the one-invocation contract, transport options, environment variables, GitHub Action inputs, and result behavior.

Transports

opencode

Use this for real OpenCode runtime behavior, including project-local agents, skills, plugins, and tool assertions.

Legacy authentication compatibility is seeded from the normal OpenCode auth file:

~/.local/share/opencode/auth.json

OpenCode V2 provider connections are stored in the normal OpenCode database:

~/.local/share/opencode/opencode.db

The runner does not mount that database directly. It creates a temporary schema-only copy containing only the credential rows and migration journals; session, project, message, event, and other runtime tables remain empty. That sanitized credential database is mounted read-only and copied into the container's fresh writable data directory.

The runner pins the catalog through OpenCode's supported registry-file interface. For OpenCode invocations, it uses an explicit --models-catalog path first, then OPENCODE_EVAL_RUNNER_MODELS, and otherwise downloads the current registry from https://models.opencode.ai/api.json before launching the container. It validates and snapshots the complete source, then mounts it read-only at /seed/models.json.

OpenCode receives OPENCODE_MODELS_PATH=/seed/models.json and OPENCODE_DISABLE_MODELS_FETCH=1; the runner does not use or discover the legacy XDG model cache. The auth and sanitized V2 credential database are copied into fresh container-local XDG directories. Sessions, history, projects, the rest of the cache, and the global config tree are not inherited.

An OpenCode config is not inherited automatically. The container constructs a minimal config unless you explicitly pass one with --config or OPENCODE_EVAL_RUNNER_CONFIG.

This avoids importing host MCPs, plugins, agent defaults, or provider overrides into behavioral evidence. Only explicit seed files are mounted; the host OpenCode config directory, data directory, cache, and sessions are never mounted.

Each invocation starts a fresh OpenCode process and consumes the pinned registry file before model selection. Catalog acquisition or validation failure stops preparation before inference; there is no bundled/legacy-cache fallback. Built-in eval acquires once and shares the same snapshot across its phases, iterations and retries. External profiles retain their per-spec catalog choices.

Known API-key environment variables are passed when present:

• OPENAI_API_KEY
• ANTHROPIC_API_KEY
• OPENROUTER_API_KEY

Additional variables require explicit --env NAME.

Reasoning can be pinned explicitly with --reasoning LEVEL. The pinned stock OpenCode 2.2.2 CLI represents a model variant in the model reference, so the runner maps --model provider/model --reasoning LEVEL to opencode run --model provider/model#LEVEL. Supplying both a #variant in --model and --reasoning is rejected as ambiguous. If the model reference already contains a variant and --reasoning is omitted, the result records that variant with "reasoning_source": "model-variant". If neither form supplies a level, the runner leaves OpenCode's provider/model default untouched and records "reasoning": "provider-default".

github-copilot-cli

Use this for pure model/role/judge execution when OpenCode runtime tools are not required.

Authentication precedence for container execution is:

1. COPILOT_GITHUB_TOKEN
2. GH_TOKEN
3. GITHUB_TOKEN
4. host gh auth token fallback

When no token environment variable is set, the host wrapper uses an authenticated GitHub CLI session if gh is available. The token is injected into the container as COPILOT_GITHUB_TOKEN via the child-process environment; its value is not written to disk or placed on the command line.

The container uses fresh COPILOT_HOME and COPILOT_CACHE_HOME, disables auto-update, prompt-mode extensions, repo hooks, MCPs, remote operations, and model tools.

Reasoning can be pinned with the same --reasoning LEVEL runner option. Copilot maps this to its native --effort LEVEL. The runner forwards the requested value exactly and never substitutes another level; acceptance or rejection of a model/effort combination remains the pinned Copilot CLI's responsibility and is surfaced through its normal exit status/output. When omitted, the result records "reasoning": "provider-default". Results also include "reasoning_source": "explicit" or "provider-default" so consumers do not confuse a requested level with an inferred provider default.

This transport reuses the trust-boundary pattern already proven in nrkno/mats-opencode-setup.

Local usage

For the generic eval interface, see Public eval CLI and profile reference. For one-invocation options and defaults, see Invocation usage and interface reference.

Built-in skill evals can combine explicitly named input files. Each --input is paired by position with one --skill; there is no directory scan or input discovery. For example, from the evaluated project root:

export OPENCODE_EVAL_RUNNER_OPENCODE_IMAGE='<compatible-full-local-image-id-or-digest-qualified-image>'

opencode-eval-runner eval \
  --input skills/python-testing/evals/evals.json --skill python-testing \
  --input skills/python-error-handling/evals/evals.json --skill python-error-handling \
  --family skill --all --iterations 5 \
  --target-model '<provider/model>' --format markdown

Built-in execution requires a compatible immutable image already available locally; the runner never pulls it. This example selects every submitted case and runs five iterations per case, separately grouped and compared within each skill. Markdown and HTML show the run → skill → case → iteration hierarchy; JSON and YAML retain their result semantics. See the eval reference for input binding, format selection, and rendering a saved run.

Build the transport you need:

podman build -f Containerfile --target opencode -t opencode-eval-runner:opencode-local .
podman build -f Containerfile --target copilot -t opencode-eval-runner:copilot-local .

The published runtime images are split by transport and do not contain Node/npm. The build stages may use Node or curl to obtain the pinned native executables, but only the native binary and the minimal Python runtime land in the final image.

Prepare a prompt:

printf '%s\n' 'Respond with the production decision for this scenario.' > /tmp/prompt.txt

Run a real OpenCode invocation:

PYTHONPATH=. python3 bin/opencode-eval-runner invoke \
  --engine podman \
  --image opencode-eval-runner:opencode-local \
  --transport opencode \
  --workspace /path/to/evaluated/project \
  --model openai/gpt-5.3-codex-spark \
  --reasoning medium \
  --agent reviewer \
  --prompt-file /tmp/prompt.txt \
  --output /tmp/target.json

The wrapper detects the normal OpenCode auth.json and opencode.db when present. Catalog selection follows the explicit path > OPENCODE_EVAL_RUNNER_MODELS > current registry download precedence. The database is always sanitized before it enters the container.

Container networking uses the OCI engine's normal default unless explicitly overridden. For hosts where rootless Podman DNS is broken, an opt-in override can be supplied:

PYTHONPATH=. python3 bin/opencode-eval-runner invoke \
  --engine podman \
  --network host \
  --transport opencode \
  --workspace /path/to/evaluated/project \
  --model openai/gpt-5.5 \
  --agent general \
  --prompt-file /tmp/prompt.txt \
  --output /tmp/target.json

--network accepts the OCI engine's network mode or network name, such as host, bridge, slirp4netns, or a custom network. It is never enabled implicitly. In particular, host weakens network isolation by sharing the host network namespace and should be used only when explicitly required.

Override the source files explicitly as needed:

--auth /path/to/auth.json
--database /path/to/opencode.db
--models-catalog /path/to/models.json
--config /path/to/opencode.json

or:

OPENCODE_EVAL_RUNNER_AUTH=/path/to/auth.json
OPENCODE_EVAL_RUNNER_DB=/path/to/opencode.db
OPENCODE_EVAL_RUNNER_MODELS=/path/to/models.json
OPENCODE_EVAL_RUNNER_CONFIG=/path/to/opencode.json

Requiring a runtime plugin

Runtime evals can fail closed before inference when a specific plugin namespace is load-bearing:

opencode-eval-runner invoke \
  --transport opencode \
  --expected-plugin loom \
  --agent general \
  ...

The expected-plugin preflight is zero-inference. The runner requires the plugin entrypoint to be materialized in the isolated config, starts a private stock OpenCode 2.2.2 server, creates a non-resuming Session prompt so plugin activation reaches its barrier without model inference, and verifies that the named plugin is present and active in the runtime plugin inventory. Agent resolution remains part of the real opencode run --agent invocation. Missing materialization, activation failure, or later agent resolution failure is infrastructure/non-evidence, never a behavioral FAIL. Actual tool use remains a repository-owned behavioral assertion in the eval corpus.

Evaluating a skill

Skills are evaluated through a normal OpenCode agent, not as a separate transport. Use --skill to identify the skill under test while keeping the prompt and grading semantics in the eval repository:

PYTHONPATH=. python3 bin/opencode-eval-runner invoke \
  --engine podman \
  --transport opencode \
  --workspace /path/to/evaluated/project \
  --model openai/gpt-5.5 \
  --agent reviewer \
  --skill architectural-design \
  --prompt-file /tmp/prompt.txt \
  --output /tmp/skill-target.json

--skill does not force-load or inject the skill into the prompt. OpenCode discovers skills normally from the evaluated workspace/config, and the agent must load the skill through the native skill tool. The result records both the requested skill and normalized skills_loaded, derived from observed tool actions. This lets the repository-owned harness prove that the intended skill was actually used without moving PASS/FAIL policy into the runner.

The runner accepts both the older {"name":"..."} and V2 {"id":"..."} skill-tool argument shapes when building skills_loaded.

Copilot CLI locally

# Either export a supported token, or authenticate the host GitHub CLI:
gh auth status

PYTHONPATH=. python3 bin/opencode-eval-runner invoke \
  --engine podman \
  --image opencode-eval-runner:copilot-local \
  --transport github-copilot-cli \
  --workspace /path/to/evaluated/project \
  --model gpt-5.4 \
  --prompt-file /tmp/prompt.txt \
  --system-file /tmp/system.txt \
  --output /tmp/judge.json

The token value is not placed on the container command line.

GitHub Actions

Pull requests expose two stable CI checks:

• test: fast Python/unit checks plus the fake Copilot GitHub Action smoke test.
• runtime-integration: stock-runtime probes and acceptance checks when runtime-relevant files change. For documentation-only or other unrelated changes, the job still runs and succeeds with an explicit not-applicable result.

Publish images remains a post-merge/tag workflow and is not a required pull-request check.

For all 21 Action inputs, defaults, execution-mode precedence, and CLI-only capabilities, see Invocation usage and interface reference.

The repository is a composite GitHub Action. It supports either a single direct invocation or setup plus a repository-owned eval harness.

For a direct Copilot invocation, the action uses the workflow's built-in GITHUB_TOKEN; no separate Copilot secret is needed. The calling workflow must grant copilot-requests: write:

permissions:
  contents: read
  copilot-requests: write

steps:
  - uses: actions/checkout@v4

  - uses: bateau84/opencode-eval-runner@main
    with:
      engine: docker
      prepare-opencode: false
      transport: github-copilot-cli
      model: gpt-5.4
      workspace: .
      prompt-file: .github/evals/prompt.txt
      output: .opencode-evals/copilot-result.json

For an OpenCode skill eval, identify the skill explicitly while still letting the evaluated agent decide whether to load it:

  - uses: bateau84/opencode-eval-runner@main
    with:
      engine: docker
      prepare-copilot: false
      transport: opencode
      model: openai/gpt-5.5
      agent: reviewer
      skill: architectural-design
      prompt-file: .github/evals/architecture-review.txt
      output: .opencode-evals/architectural-design.json

Larger suites can use the action as the execution boundary and run their own corpus harness:

  - uses: bateau84/opencode-eval-runner@main
    with:
      engine: docker
      command: |
        python3 scripts/run-evals.py \
          --cases INTENT-01,WORK-01,REVIEW-01,CRITIC-01

The action exposes the workflow's built-in GITHUB_TOKEN to both direct Copilot invocations and repository-owned commands. In Action mode it deliberately clears inherited COPILOT_GITHUB_TOKEN and GH_TOKEN values so Copilot cannot silently select a higher-precedence credential. The runner passes only the selected token variable name into the isolated Copilot container; the token value is not placed on the command line.

For OpenCode credentials in CI, materialize a protected secret as a file before the eval and point OPENCODE_EVAL_RUNNER_AUTH at it. Do not commit auth files.

Example:

- name: Materialize OpenCode auth
  shell: bash
  env:
    OPENCODE_AUTH_JSON: ${{ secrets.OPENCODE_AUTH_JSON }}
  run: |
    install -m 700 -d "$RUNNER_TEMP/opencode-auth"
    printf '%s' "$OPENCODE_AUTH_JSON" > "$RUNNER_TEMP/opencode-auth/auth.json"
    chmod 600 "$RUNNER_TEMP/opencode-auth/auth.json"
    echo "OPENCODE_EVAL_RUNNER_AUTH=$RUNNER_TEMP/opencode-auth/auth.json" >> "$GITHUB_ENV"

Result contract

Each invocation writes one opencode-eval-runner/v1 JSON document. Every official result includes the canonical runtime_evidence object.

{
  "schema": "opencode-eval-runner/v1",
  "transport": "opencode",
  "model": "openai/gpt-5.3-codex-spark",
  "reasoning": "medium",
  "reasoning_source": "explicit",
  "agent": "reviewer",
  "skill": "architectural-design",
  "exit_code": 0,
  "session_id": "...",
  "text": "...",
  "tools": [],
  "actions": [],
  "skills_loaded": [],
  "stderr": "",
  "stdout": "...",
  "runtime_evidence": {
    "schema": "opencode-eval-runner/runtime-evidence/v1",
    "status": "complete",
    "evidence_eligible": true,
    "observations": [],
    "coverage": {
      "observation_closed": {"state": "available", "value": true},
      "process_state": "completed",
      "starts": {"state": "available", "value": 0},
      "terminals": {"state": "available", "value": 0},
      "missing_terminals": {"state": "available", "value": 0},
      "observer_failures": {"state": "available", "value": 0},
      "callback_failures": {"state": "available", "value": 0},
      "losses": [],
      "unsupported": ["stock_codemode_final_boundary_not_exposed"],
      "boundaries": {
        "native": {
          "status": "complete",
          "evidence_eligible": true,
          "starts": {"state": "available", "value": 0},
          "terminals": {"state": "available", "value": 0},
          "missing_terminals": {"state": "available", "value": 0},
          "issues": []
        },
        "code_mode_execution": {
          "status": "complete",
          "evidence_eligible": true,
          "starts": {"state": "available", "value": 0},
          "terminals": {"state": "available", "value": 0},
          "missing_terminals": {"state": "available", "value": 0},
          "issues": []
        },
        "code_mode_finality": {
          "status": "unsupported",
          "evidence_eligible": false,
          "starts": {"state": "unsupported", "reason": "stock_codemode_final_boundary_not_exposed"},
          "terminals": {"state": "unsupported", "reason": "stock_codemode_final_boundary_not_exposed"},
          "missing_terminals": {"state": "unsupported", "reason": "stock_codemode_final_boundary_not_exposed"},
          "issues": ["stock_codemode_final_boundary_not_exposed"]
        }
      }
    }
  }
}

The container emits this object as a single JSON line on stdout. The host harness re-validates runtime_evidence before it writes the result artifact, so a missing or malformed runtime-evidence object is rejected rather than silently downgraded.

Large OpenCode model catalogs

The runner treats models.json as a potential credential source, not harmless metadata. The credential inventory scans up to 64 MB of model-catalog JSON while retaining the original 4 MB bound for auth/config. It reads and validates the entire selected source, including sensitive values near its end; no prefix-only, skipped-source, or trust-by-extension exceptions are permitted.

Oversized, malformed, or unreadable catalogs fail host-side preflight before inference begins with an actionable error. The container independently repeats the fail-closed scan before output: if its inventory is unavailable, required observations remain non-evidence and the result-sink guard redacts convenience fields instead of reporting unsanitized data. The higher catalog-specific bound is deliberate and finite to avoid unbounded memory use. If the catalog exceeds it, supply a valid bounded --models-catalog snapshot.

The runtime image and host runner must be updated together; changing only the host Python library does not change the container's credential scanner.

runtime_evidence is required for every official transport result. github-copilot-cli also emits the canonical object, but with status: "unsupported" because it has no OpenCode runtime observer.

If you override --image, treat the host runner and image as one compatibility pair. Legacy or custom images that do not emit a valid opencode-eval-runner/runtime-evidence/v1 object are rejected by this host version; upgrade the host executable and image together.

For runtime verdicts, do not use top-level evidence_eligible by itself. An assertion may use evidence only when every boundary it requires is complete and every exact field it requires is available. redacted, omitted, or unsupported required fields are not PASS evidence. Existing tools, actions, tool_result_evidence, stdout/stderr, and model text are diagnostic/convenience data and must not fill an authoritative-evidence gap. See Runtime evidence contract v1.

The eval repository still owns the assertion semantics and decides PASS, FAIL, or non-evidence after applying those eligibility rules.

Image versions

The transport images currently pin:

• OpenCode CLI 2.2.2
• GitHub Copilot CLI 1.0.83

The two CLIs are not bundled together. OpenCode's npm package is used only as a build-time native-binary selector; GitHub Copilot CLI is installed from its native release installer. Node/npm are absent from the final runtime images.

Pinning is deliberate: behavioral evidence should not silently change because a CLI auto-updated.

main publishes transport-specific images:

ghcr.io/bateau84/opencode-eval-runner:opencode-edge
ghcr.io/bateau84/opencode-eval-runner:copilot-edge
ghcr.io/bateau84/opencode-eval-runner:opencode-sha-...
ghcr.io/bateau84/opencode-eval-runner:copilot-sha-...

The host runner selects the correct image from --transport. Override either image with:

OPENCODE_EVAL_RUNNER_OPENCODE_IMAGE=...
OPENCODE_EVAL_RUNNER_COPILOT_IMAGE=...

Built-in eval --input ... execution is stricter: OPENCODE_EVAL_RUNNER_OPENCODE_IMAGE is required and must name a full local image ID or digest-qualified identity. The runner checks its OpenCode version and invocation-adapter labels using local engine metadata only; it does not pull or contact a registry, and each target/judge run uses pull-never. Tags and missing/incompatible local images are rejected. Existing --profile image resolution remains unchanged.

Tags matching v* are published with opencode- and copilot- prefixes.

Evaluation trust model

The normal evaluation profile is a trusted-checkout profile. It assumes the runner, pinned stock OpenCode runtime, reviewed instrumentation, and explicitly selected evaluated checkout/dependencies are trusted components of the evaluation environment.

They are not trusted merely because they produce data that looks like evidence. Model prose, tool-returned collector-shaped JSON, target-writable files, requested actions, inferred identities, and reconstructed results do not establish that an event occurred.

Authoritative runtime observations must come from reviewed instrumentation observing actual execution. Missing, partial, ambiguous, or unsupported required observations are non-evidence and must fail closed for the affected assertion.

This profile does not claim resistance to an evaluated plugin that deliberately compromises the trusted runtime or instrumentation. Hostile-plugin isolation is a separate optional profile, not a prerequisite for normal Loom evaluation.

See Trusted-checkout runtime evidence and the versioned runtime-evidence result contract.

OpenCode results now expose opencode-eval-runner/runtime-evidence/v1 as the single authoritative runtime-evidence object. Native calls are observed through the stock-2.2.2 decoded-execution and Session terminal boundaries. Code Mode inner identity/input/ordering is observable, while exact final script-visible value/error remains explicitly unsupported. Existing tools, actions, tool_result_evidence, stdout/stderr, and model text remain convenience/diagnostic data only; they do not expose runtime-evidence eligibility and are never substitutes for runtime_evidence.

Overall evidence eligibility is separate from assertion eligibility: an unsupported Code Mode finality boundary does not invalidate an unrelated complete native assertion, and redacted/omitted fields only block assertions that require those exact values.

The github-copilot-cli transport has no OpenCode runtime observer. It still emits the canonical runtime_evidence object, but with status unsupported.

Stock OpenCode 2.2.2 does not expose a supported boundary that proves the exact final value/error seen by a Code Mode script for each inner call. That assertion is reported as unsupported.

Security boundary

The runner:

• drops Linux capabilities;
• only adds extra bind mounts when the caller explicitly supplies --mount SOURCE:TARGET[:ro|rw];
• enables no-new-privileges;
• uses a read-only container root filesystem with ephemeral /tmp;
• mounts the evaluated workspace read-only unless --workspace-mode rw is explicitly selected;
• mounts config/auth/model-catalog seed files read-only;
• on rootless Podman, disables SELinux container labeling instead of relabeling the user's repository/auth files;
• creates fresh OpenCode/Copilot state per invocation;
• passes only explicit credential environment variables;
• never treats infrastructure/provider failure as behavioral evidence.

This is an eval isolation boundary, not a sandbox for hostile arbitrary code. A deliberately malicious evaluated plugin running inside an invocation still has the network and credentials granted to that invocation.