eval(headless): DeepSeek V4 Flash Terminal-Bench 2.1 harness A/B — Maka vs OpenCode #1680

Goal

Produce reproducible evidence of whether Maka or OpenCode is more effective and resource-efficient when both use DeepSeek V4 Flash under the same Terminal-Bench 2.1 conditions.

This is a same-model harness comparison, not a model bake-off.

Frozen experiment design

• Benchmark: Terminal-Bench 2.1, all 89 tasks
• Task revision: d49e28f1e4ddd13d289e85a5f312a66750951932
• Task-tree fingerprint: sha256:456826aa4c47ed309716c964c96d2a3acc998764ebc84f3e8449c807d74bd4e7
• Arms: Maka vs OpenCode 1.17.18
• Model: deepseek/deepseek-v4-flash
• Reasoning effort: max
• Attempt policy: paired Pass@1
• Execution policy: up to four task pairs concurrently, with Maka and OpenCode started in parallel inside each pair; at most eight cells may run concurrently
• Both arms use the same model, provider credential, task order, task instruction, deadline, verifier, and retry policy while retaining their native harness behavior.
• Retry only infrastructure-invalid cells. Agent failures remain benchmark outcomes.
• Credentials stay behind the host-side provider proxy and must not enter task containers or published artifacts.
• The experiment configuration is frozen from the first accepted full-v5 result onward.

Current run

• Run ID: deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v5
• Started: 2026-07-31
• Status: full 89-pair run in progress
• Pair concurrency: 4
• Maximum concurrent cells: 8
• Arm execution: parallel
• The immutable manifest and append-only run journal have been created.
• Runs v1–v4 are preserved as superseded or infrastructure-invalid evidence and are excluded from the accepted v5 comparison.
• No external Oracle registry snapshot is configured. Oracle evidence is advisory; the official Terminal-Bench verifier remains the scoring authority.

Execution plan

1. Run all 89 task pairs under the frozen v5 manifest.
2. Continuously audit verifier validity, execution identity, provider usage, tool settlement, and infrastructure failures.
3. Preserve invalid and superseded evidence; rerun only cells demonstrated to be infrastructure-invalid.
4. Validate that paired results, usage accounting, JSON, CSV, and Markdown artifacts agree.
5. Publish a report under docs/eval/.

Reported outcomes

Effectiveness:

• Pass count and pass rate by arm
• Paired wins, losses, and ties
• Paired uncertainty and significance
• Deadline and budget-exhaustion outcomes
• Failure classification for discordant pairs

Resource efficiency:

• Uncached input, cached input, output, and total tokens
• Cache hit rate where available
• API-equivalent cost using the pricing identity captured by the manifest
• Wall-clock duration
• Tokens, cost, and duration per pass

Completion criteria

• Exactly 89 accepted task pairs are evaluated.
• Every accepted cell has valid execution identity and verifier evidence.
• JSON, CSV, and Markdown artifacts agree.
• The final report records the frozen experiment identity and known limitations.
• No credential appears in git, logs, commands, screenshots, fixtures, or published artifacts.

Related

• eval(headless): harness A/B remaining arms — GLM-5.2@TB vs OpenCode + Kimi@DeepSWE-30 #1531 — existing harness A/B evaluation arms
• Revisit default Terminal-Bench flow: light execution for strong models, heavy scaffold as diagnostic #708 — strong-model Terminal-Bench execution profile
• feat(headless): add a resumable Maka vs OpenCode GLM-5.2 benchmark #861 — original resumable Maka vs OpenCode benchmark implementation
• fix(headless): record OpenCode prompt provenance accurately #1563 — OpenCode prompt-provenance accuracy; scores and usage are unaffected, but this run must not describe the recorded Maka prompt hash as OpenCode's native system-prompt hash