docs: publish DeepSeek Flash Terminal-Bench comparison

docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.csv

@@ -0,0 +1,90 @@
+task_id,maka,opencode,maka_failure_class,opencode_failure_class
+adaptive-rejection-sampler,failed,failed,budget_exhausted,budget_exhausted
+bn-fit-modify,passed,failed,,verification_failed
+break-filter-js-from-html,passed,passed,,
+build-cython-ext,passed,passed,,
+build-pmars,passed,passed,,
+build-pov-ray,passed,passed,,
+caffe-cifar-10,passed,failed,,budget_exhausted
+cancel-async-tasks,failed,failed,verification_failed,verification_failed
+chess-best-move,failed,failed,verification_failed,budget_exhausted
+circuit-fibsqrt,passed,passed,,
+cobol-modernization,passed,passed,,
+code-from-image,passed,failed,,budget_exhausted
+compile-compcert,failed,passed,budget_exhausted,
+configure-git-webserver,passed,passed,,
+constraints-scheduling,passed,passed,,
+count-dataset-tokens,passed,passed,,
+crack-7z-hash,passed,passed,,
+custom-memory-heap-crash,passed,passed,,
+db-wal-recovery,passed,failed,,budget_exhausted
+distribution-search,passed,passed,,
+dna-assembly,failed,failed,verification_failed,verification_failed
+dna-insert,failed,passed,verification_failed,
+extract-elf,failed,failed,verification_failed,verification_failed
+extract-moves-from-video,failed,failed,budget_exhausted,budget_exhausted
+feal-differential-cryptanalysis,passed,passed,,
+feal-linear-cryptanalysis,passed,passed,,
+filter-js-from-html,failed,failed,verification_failed,verification_failed
+financial-document-processor,passed,passed,,
+fix-code-vulnerability,passed,passed,,
+fix-git,passed,failed,,verification_failed
+fix-ocaml-gc,passed,passed,,
+gcode-to-text,failed,failed,budget_exhausted,budget_exhausted
+git-leak-recovery,passed,passed,,
+git-multibranch,passed,passed,,
+gpt2-codegolf,failed,failed,budget_exhausted,budget_exhausted
+headless-terminal,passed,passed,,
+hf-model-inference,passed,passed,,
+install-windows-3.11,failed,failed,budget_exhausted,budget_exhausted
+kv-store-grpc,passed,passed,,
+large-scale-text-editing,passed,passed,,
+largest-eigenval,failed,failed,budget_exhausted,budget_exhausted
+llm-inference-batching-scheduler,passed,passed,,
+log-summary-date-ranges,passed,passed,,
+mailman,passed,passed,,
+make-doom-for-mips,failed,failed,budget_exhausted,budget_exhausted
+make-mips-interpreter,failed,failed,budget_exhausted,budget_exhausted
+mcmc-sampling-stan,passed,passed,,
+merge-diff-arc-agi-task,passed,passed,,
+model-extraction-relu-logits,passed,failed,,budget_exhausted
+modernize-scientific-stack,passed,passed,,
+mteb-leaderboard,failed,failed,verification_failed,budget_exhausted
+mteb-retrieve,passed,failed,,verification_failed
+multi-source-data-merger,passed,passed,,
+nginx-request-logging,passed,passed,,
+openssl-selfsigned-cert,passed,passed,,
+overfull-hbox,failed,failed,verification_failed,verification_failed
+password-recovery,failed,passed,budget_exhausted,
+path-tracing,passed,failed,,budget_exhausted
+path-tracing-reverse,failed,failed,budget_exhausted,budget_exhausted
+polyglot-c-py,passed,passed,,
+polyglot-rust-c,passed,passed,,
+portfolio-optimization,passed,passed,,
+protein-assembly,failed,failed,verification_failed,verification_failed
+prove-plus-comm,passed,passed,,
+pypi-server,passed,passed,,
+pytorch-model-cli,passed,passed,,
+pytorch-model-recovery,passed,passed,,
+qemu-alpine-ssh,failed,failed,budget_exhausted,budget_exhausted
+qemu-startup,failed,passed,budget_exhausted,
+query-optimize,passed,passed,,
+raman-fitting,failed,failed,verification_failed,budget_exhausted
+regex-chess,passed,failed,,budget_exhausted
+regex-log,passed,passed,,
+reshard-c4-data,passed,passed,,
+rstan-to-pystan,failed,failed,budget_exhausted,runtime_error
+sam-cell-seg,passed,passed,,
+sanitize-git-repo,passed,failed,,verification_failed
+schemelike-metacircular-eval,passed,failed,,verification_failed
+sparql-university,passed,failed,,verification_failed
+sqlite-db-truncate,passed,passed,,
+sqlite-with-gcov,passed,failed,,budget_exhausted
+torch-pipeline-parallelism,failed,failed,verification_failed,budget_exhausted
+torch-tensor-parallelism,failed,failed,verification_failed,verification_failed
+train-fasttext,failed,failed,budget_exhausted,budget_exhausted
+tune-mjcf,passed,failed,,budget_exhausted
+video-processing,failed,failed,verification_failed,verification_failed
+vulnerable-secret,passed,passed,,
+winning-avg-corewars,passed,failed,,verification_failed
+write-compressor,passed,failed,,budget_exhausted

docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md

@@ -0,0 +1,143 @@
+# Terminal-Bench 2.1 — DeepSeek V4 Flash: Maka vs OpenCode
+
+This report compares Maka and OpenCode as two agent harnesses around the same DeepSeek V4 Flash model on all 89 Terminal-Bench 2.1 tasks. It records the frozen experiment, accepted outcomes, paired inference, outcome-normalized economics, and the recovery decisions required to produce a fully scored dataset.
+
+**Run id:** `deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7`
+
+**Local artifacts (git-excluded):** `~/.maka/eval/runs/deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7/`
+
+**Metric:** end-to-end pass@1 by the official task verifier
+
+**Status:** `completed` — 178/178 cells model-scored, no unscored infrastructure cell, and no missing final usage
+
+**Per-task outcomes:** [`terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.csv`](./terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.csv)
+
+## TL;DR
+
+- **Maka passed 61/89 tasks (68.54%); OpenCode passed 49/89 (55.06%).** Maka led by 12 tasks, or 13.48 percentage points.
+- The paired outcomes were 16 Maka-only passes, 4 OpenCode-only passes, and 69 ties. An exact two-sided McNemar test on the 20 discordant pairs gives **p = 0.0118**. This is evidence about the fixed 89-task Terminal-Bench 2.1 suite, not proof of a universal advantage on other task distributions.
+- The observed accepted-dataset API-equivalent cost per pass was effectively the same: **$0.03172 for Maka and $0.03171 for OpenCode**. No cost-equivalence test was performed, so this is a descriptive point estimate rather than a statistical equivalence claim.
+- Two diagnostics accompany, but do not replace, end-to-end pass@1. On the shared 61-pair subset obtained by excluding any pair where either arm exhausted its budget, Maka passed 52/61 (85.25%) and OpenCode 46/61 (75.41%). Budget exhaustion affected 15/89 Maka cells (16.85%) and 24/89 OpenCode cells (26.97%). These observations decompose the final gap; this run does not identify their causal mechanisms.
+
+## Results
+
+End-to-end pass@1 is the primary result. Budget-exhausted cells remain scored failures in its denominator.
+
+| Primary result | Maka | OpenCode | Maka − OpenCode |
+| --- | ---: | ---: | ---: |
+| End-to-end pass@1 | **61/89 (68.54%)** | **49/89 (55.06%)** | **+12 tasks (+13.48 pp)** |
+
+The paired outcome table is:
+
+| | OpenCode pass | OpenCode fail | Total |
+| --- | ---: | ---: | ---: |
+| Maka pass | 45 | 16 | 61 |
+| Maka fail | 4 | 24 | 28 |
+| Total | 49 | 40 | 89 |
+
+For the exact McNemar test, the null assigns equal probability to either direction among discordant pairs. With 16 Maka-only and 4 OpenCode-only outcomes, the two-sided exact binomial probability is `2 × P[Binomial(20, 0.5) ≤ 4] = 0.0118179`. The test treats the 89 benchmark tasks as the paired units. Because this was one frozen run over one fixed suite, the p-value should not be read as a guarantee for other benchmarks, task distributions, provider conditions, or repetitions.
+
+## Diagnostic decomposition
+
+The following metrics diagnose where the observed end-to-end gap appears; neither is an alternate headline score.
+
+| Diagnostic | Maka | OpenCode | Maka − OpenCode |
+| --- | ---: | ---: | ---: |
+| Non-budget Conditional Pass Rate | 52/61 (85.25%) | 46/61 (75.41%) | +9.84 pp |
+| Budget Exhaustion Rate | 15/89 (16.85%) | 24/89 (26.97%) | −10.11 pp |
+
+The conditional denominator excludes the entire pair whenever either arm is `budget_exhausted`, leaving the same 61 tasks for both arms. It is not an “unlimited-time pass rate”: the remaining tasks still ran under their original budgets, and the excluded set is not random. The simultaneous conditional-pass and budget-exhaustion differences show that both dimensions accompany the end-to-end result. They do not by themselves isolate model solution quality, inference speed, or agent turnover efficiency as causes.
+
+## Outcome-normalized economics
+
+Cost per pass is the primary economic denominator because it normalizes recorded usage by successful benchmark outcomes. It includes spending on all accepted scored failures, but excludes superseded infrastructure-invalid admissions from the accepted dataset.
+
+| Economic result | Maka | OpenCode |
+| --- | ---: | ---: |
+| Accepted-dataset API-equivalent cost per pass | **$0.031718** | **$0.031712** |
+| Passed tasks | 61 | 49 |
+
+The observed difference is less than $0.000007 per pass, or 0.02%. At the precision supported by this single run, the two point estimates are effectively the same. This is not a statistical cost-equivalence result: the experiment has one repetition and no uncertainty model for cost per pass.
+
+For completeness, the resource footprint behind that outcome-normalized result is:
+
+| Resource diagnostic | Maka | OpenCode |
+| --- | ---: | ---: |
+| Accepted scored-cell cost | $1.934813 | $1.553888 |
+| Cost per scored cell | $0.021739 | $0.017459 |
+| Total tokens | 173,281,686 | 115,523,680 |
+| Cached input tokens | 166,618,240 | 110,226,048 |
+| Uncached input tokens | 3,315,719 | 2,083,315 |
+| Output tokens | 3,347,727 | 3,214,317 |
+| Cache-hit share of input | 98.05% | 98.15% |
+
+OpenCode had a lower per-cell footprint, but it also produced fewer passes. The lower aggregate spend therefore does not become an observed cost-per-success advantage. The report keeps resource footprint and outcome-normalized economics separate rather than combining them into a composite score.
+
+Costs are cache-aware API-equivalent estimates from the pricing identity frozen in the manifest: $0.145 per million uncached input tokens, $0.0029 per million cache-hit input tokens, and $0.29 per million output tokens. They are not a billing invoice.
+
+## Frozen setup
+
+| Dimension | Value |
+| --- | --- |
+| Benchmark | Terminal-Bench 2.1, revision `d49e28f1e4ddd13d289e85a5f312a66750951932`; all 89 tasks |
+| Task-tree fingerprint | `sha256:456826aa4c47ed309716c964c96d2a3acc998764ebc84f3e8449c807d74bd4e7` |
+| Run fingerprint | `sha256:d18bc4c5ab0dd15161f48838ab0fa4a7a9ba2654f6008afe36d3d398fdf5fff3` |
+| Model | `deepseek-v4-flash` through the DeepSeek provider on both arms |
+| Reasoning effort | `max` on both arms |
+| Repetitions | 1 |
+| Metric | Paired pass@1 |
+| Attempt policy | One accepted model attempt per arm/task; only infrastructure-invalid admissions may be replaced |
+| Deadline policy | Task-native agent timeout ×1; 900-second outer setup and teardown grace |
+| Pair execution | Up to four task pairs concurrently; Maka and OpenCode start in parallel within a pair; at most eight cells concurrently |
+| External system prompt | Empty on both arms |
+| Maka arm | `maka_agent:MakaAgent`; continuation off; active and stale tool-result pruning enabled at a 2,048 estimated-token threshold; semantic compact off |
+| OpenCode arm | `opencode_agent:MakaOpenCodeAgent` 1.17.18; pure mode; automatic permissions; `max` variant |
+| Billing mode | Metered, using the frozen DeepSeek V4 Flash pricing identity above |
+
+This is a same-model harness comparison, not an isolated model evaluation and not a same-system/same-tool ablation. The two arms retain their native agent instructions, tool behavior, context management, and execution loops. The observed difference is therefore attributable to the compared harness systems as a whole; this run does not rank individual harness differences as causes.
+
+## Outcome accounting and recovery
+
+The accepted dataset selects the latest authorized terminal outcome for each arm/task cell under the immutable run fingerprint. The attempts WAL contains **183 Agent admissions**: 178 initial admissions plus five single infrastructure retries. Those retries covered four Maka cells (`winning-avg-corewars`, `polyglot-rust-c`, `path-tracing`, and `qemu-startup`) and one OpenCode cell (`polyglot-rust-c`). No cell received a third Agent admission.
+
+The adjudication ledger contains 25 taxonomy-only corrections that did not change pass/fail: 24 OpenCode cells with authoritative Harbor `AgentTimeoutError` evidence were normalized from `verification_failed` to scored `budget_exhausted`, and OpenCode `rstan-to-pystan` was normalized from harness infrastructure to a scored candidate runtime/resource failure after substantial execution and an official verifier result. Two pre-execution Maka failures were reclassified as unscored infrastructure before receiving their one allowed replacement admission.
+
+The final OpenCode `polyglot-rust-c` admission completed all five provider requests, recorded complete usage, wrote and locally compiled the requested candidate, and then encountered verifier setup infrastructure failure twice because the task container could not reach the Ubuntu archive. The container was deleted, but the single-file candidate was durably and completely represented in the second admission's trajectory. The candidate was reconstructed byte-for-byte (`sha256:dfa4c4cd3b41744e6fd4f971ca544075cc305d625bcd4b515b8ba559438c2cd3`) and replayed only through the unchanged official verifier in the pinned `linux/amd64` task image with the original 1 CPU / 2 GiB limits. The official test passed and returned reward 1. This recovery changed the cell from unscored infrastructure to pass without invoking the model again; the Agent admission count remained two.
+
+The recovery layer is append-only: superseded events remain in the WAL, adjudications identify their source event, and the final verifier-only replay has a separate local evidence manifest. The committed CSV contains only the selected final pass/fail and failure-class projections; prompts, trajectories, provider payloads, raw logs, and local capabilities are not published.
+
+## Caveats
+
+- This is one repetition over a fixed 89-task suite. The exact paired p-value describes asymmetry on this suite and does not establish universal superiority.
+- The arms share the model, provider, task order, instructions, verifier, deadline policy, and infrastructure-retry policy, but they intentionally retain native harness behavior. This is not a single-variable component ablation.
+- Non-budget Conditional Pass Rate is selection-conditional and diagnostic. It cannot be interpreted as an unlimited-time counterfactual.
+- Budget exhaustion records that a valid candidate opportunity reached the benchmark deadline and then received an official verifier result. It does not by itself distinguish provider latency, model generation length, tool time, or agent policy.
+- Cost per pass is an accepted-dataset point estimate from recorded usage and frozen API-equivalent prices. It includes scored failures but excludes superseded infrastructure-invalid admissions, whose complete metering is not always available. “Effectively the same” describes the observed values, not a formal equivalence interval or total experiment-operations cost.
+- No external Oracle registry snapshot was configured. Oracle annotations are missing for all 89 tasks; the official Terminal-Bench verifier remains the scoring authority.
+- Five infrastructure-invalid Agent admissions were replaced under the same bounded policy, and one completed candidate required verifier-only replay. These recoveries are disclosed above because omitting them would make the final coverage and score non-reconstructible.
+
+## Integrity
+
+SHA-256 hashes of the frozen local evidence and committed outcome projection:
+
+| Source | SHA-256 |
+| --- | --- |
+| `harness-ab-manifest.json` | `c364730ac5e3db4ff8a6cf0b22a50b4e4f953ee05d090dc69415cb9012376e9c` |
+| `harness-ab-report.json` | `bd616091c18bed6ad8d4bfe7534c7f4d6a6861c01b1c6f383d408b39673f922b` |
+| `controller/results.jsonl` | `19764b90cbebddeb05c20d58be8760bfdabac1bf99e1c15a0528c0f934cd9835` |
+| `controller/results.jsonl.attempts.jsonl` | `4878ca5b612d20d87a28066456ddcdc32a5c0bb105940cb0e6ef270a2355c7ab` |
+| `controller/adjudications.jsonl` | `7253f3cf043cf5079428b704baf3ee30b8c259a4dc806704cb7016914a4cd667` |
+| Verifier-only replay `evidence.json` | `93b786d4305452bf1f1ef15f5474f5902bc979255e89c61f5669581caab5178b` |
+| Committed outcome CSV | `a1f52e1fbb4463ed1f9a6d591dd50aa8eaff67dbb825d05a36213d0d3a25e766` |
+
+## Artifact pointers
+
+| Artifact | Local path |
+| --- | --- |
+| Generated report | `~/.maka/eval/runs/deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7/harness-ab-report.{json,csv,md}` |
+| Immutable manifest | `~/.maka/eval/runs/deepseek-v4-flash-maka-vs-opencode-tbench-2.1-full-v7/harness-ab-manifest.json` |
+| Controller WAL | `.../controller/results.jsonl` and `.../controller/results.jsonl.attempts.jsonl` |
+| Adjudication ledger | `.../controller/adjudications.jsonl` |
+| Verifier replay evidence | `.../controller/verifier-replays/ab-opencode-r0-polyglot-rust-c-attempt-2/evidence.json` |
+
+The experiment was tracked in [issue #1680](https://github.com/maka-agent/maka-agent/issues/1680).