Repository navigation
feat(headless): add createPierTaskRunner for DeepSWE via Pier - #1352
Merged
Merged
Conversation
Add an optional fixed listen port so a caller can present the proxy on a Squid-legal port (80/443), which Pier's egress requires for offline tasks, with a clear error when a privileged or busy port cannot be bound. The default stays an ephemeral port, so existing callers are unchanged.
Implement the TaskRunner seam over Pier's CLI: spawn `pier run` per task with the maka repo bind-mounted at /opt/maka-agent, split agent env across three channels (--ae for identity/config, the pier process env for MAKA_BACKEND which the adapter CliFlag reads only from os.environ, and a 0600 --env-file for proxy-minted secrets kept off argv), map the trial dir (verifier/reward.json, agent/maka-cell-output.json, result.json) to TaskRunOutput, and classify timeouts, trial exceptions, and budget exhaustion. The Kimi arm runs the host provider auth proxy on a Squid-legal port and hands the container MAKA_PROVIDER_PROXY_URL/TOKEN; the real provider key never leaves the host. Refs #1343.
…rs stay loud server.once registered an anonymous wrapper while server.off removed the never-registered reject, so after a successful listen the stale wrapper swallowed later server socket errors into the settled bind promise. Name the handler, remove the same reference, and lock it with a listenerCount regression test.
…cess env pier's --ae parser strips leading/trailing whitespace from values, so a prompt with a trailing newline was truncated before reaching the adapter, breaking the execution-identity hash round-trip on every task. Move MAKA_SYSTEM_PROMPT to the pier process env, which both adapters read byte-exact through their os.environ fallbacks, and keep it out of --ae where a stripped extra_env copy would shadow the correct value.
The adapter's model_name takes precedence over MAKA_MODEL, so passing the provider-prefixed id (deepseek/deepseek-v4-flash) through -m leaked the prefixed id into the cell. Normalize with the same modelIdForProvider contract the Harbor runner uses.
agentEnv could silently override MAKA_MODEL, MAKA_TRIAL_* pricing, and other benchmark identity keys. Export the Harbor runner's assertNoExperimentIdentityOverrides and enforce the identical key set in the Pier runner, so the invariant cannot drift between runners.
A budget-exhausted pier trial threw FixedPromptBudgetExhaustedError with null artifactRefs, so the controller demoted every timeout to missing_execution_identity and excluded it from the Pass@1 denominator. Export the Harbor runner's readTimedOutTrialArtifacts (and harborTraceMode) and reuse them: both runners' trials are written by the same adapters into the same agent/ layout, so the recovery contract is shared by construction rather than approximated by a copy.
…launch A failure between proxy start and pier launch (env-file write, arg assembly) escaped the finally that closes the proxy, leaking the listening socket. Extend the finally to cover the whole post-start window; bind errors still surface raw because they occur before the proxy exists.
…port Every Kimi attempt binds the same fixed port (default 443), so controller maxConcurrency > 1 made the second attempt fail with EADDRINUSE. Serialize the proxy-holding section with a per-runner queue instead of sharing one proxy, preserving per-attempt usage/telemetry attribution. Squid leaves only 80/443 usable, so a port pool is deferred until concurrency needs it.
…unner Pier's EnvironmentConfig has no extra_docker_compose or platform field (verified against pier 0.3.0 models/trial/config.py), so the option could never take effect. Remove it rather than keep a dead promise; the constraint is documented on the environment option.
Astro-Han
force-pushed
the
feat/headless-pier-task-runner
branch
from
July 22, 2026 16:50
e32197b to
47055fb
Compare
…ve budget A fixed 45-minute default systematically undercut DeepSWE's native per-trial budget (5400s agent + 1800s verifier on all 113 tasks), killing every real trial mid-flight. Extract the Harbor runner's derivation as resolveNativeTrialTimeoutMs — (agent + verifier) x multiplier + setup/teardown grace, floored at 45 minutes — and reuse it, with Pier supplying the task's native verifier seconds. Explicit pierTimeoutMs still wins.
The runner returned harbor: { reward } without a verifier field, so the controller never treated pier-graded failed cells (max_tokens, tool_step_cap_reached, policy_denied) as verifierGraded — they left the benchmark denominator as scored=false. Pier's verifier grading is the scoring authority: construct the single-attempt HarborVerifierOutcome from the graded reward (which, unlike Harbor's independently written oracle artifact, cannot disagree with it). Locked by a controller-level test: max_tokens failed cell + pier reward maps to task_completed with scored=true.
… timeout Pier's trial.py records an AgentTimeoutError and then unconditionally runs verification, so a timed-out agent can still be fully graded (reward.json present) — but the runner threw FixedPromptBudgetExhaustedError unconditionally and discarded earned rewards. Mirror Harbor's completeTimedOutTrial semantics: when the grading authority and the cell output are both present, return the graded trial normally (structured verifier included); only an ungraded budget exhaustion throws, still carrying recovered artifacts. Also corrects the stale comment claiming exception_info implies the verifier never ran.
… runner Delete the pier runner's verbatim copies of providerTokenSummary (cost math), providerProxyAuthMode, providerProxyUsageProtocol, mergeAgentEnv, assertNoProviderSecretsInAgentEnv, isBudgetExhaustedError, and isBudgetExhaustedTrialException by exporting the Harbor originals — same treatment as the already-shared identity/timeout/artifact functions, closing the drift surface. PierTaskPricing becomes an alias of HarborTaskPricing. Drop the dead now option copied from Harbor. PierInfraError stays separate with the rationale written down (the controller classifies infra by behavior, not identity; diagnostics must name the failing harness), and the remaining 3-line generic utilities keep local copies under an explicit no-benchmark-semantics note.
execFile's timeout path SIGKILLed pier directly, skipping every Python finally: docker compose containers leaked, and on the Maka host-cell arm the orphaned run-host-cell.mjs kept burning real tokens until its own cell deadline. Spawn pier detached in its own process group and terminate in two phases — group SIGTERM first (pier's SIGTERM handler raises KeyboardInterrupt and its finally chain owns docker teardown), then group SIGKILL after a 120s grace sized for compose teardown. Scoring semantics unchanged (timedOut still reported). Harbor's runner shares this gap on main; its CLI signal semantics are unverified, so that fix stays out of scope here.
…lifecycle The derivation missed the environment build phase and pier's own retry policy: every DeepSWE task carries build_timeout_sec=1800, and pier retries environment start and verification once each on their timeout errors (tenacity stop_after_attempt(2), pier/trial/execution.py:208 and pier/trial/trial.py:333), so a legitimate cold-build trial can run 2xbuild + agent + 2xverifier = 12600s — above the previous 8100s budget. Parse [environment] build_timeout_sec into task metadata along the existing seam, generalize the shared derivation to take the runner-owned lifecycle seconds (Harbor behavior unchanged), and give the pier runner one function owning the complete pier lifecycle model with the pier source facts cited — locked by a contract test asserting the DeepSWE budget covers 12600s plus grace.
…OST_NO_AUTH Mirror the Harbor runner's provider predicate in pierProviderRuntime with a backend gate: backend 'fake' keeps the inert in-container cell untouched, while the default ai-sdk backend without an apiKeyFile now wires the host cell with MAKA_HOST_NO_AUTH for registry authKind 'none' providers (ollama/lm-studio/localai) instead of silently dropping the host runtime. Secret-requiring providers without a key still return no runtime so the adapter fails loud at environment creation.
Port 0 asks the OS for a fresh port per bind, so serializing those attempts only throttled throughput without preventing any collision. The lock now applies only when the Kimi arm binds a fixed port (the 443 default), where a second concurrent bind is a guaranteed EADDRINUSE.
…r Pier The cell output's runtimeEventsPath contract points at agent/runtime-events.jsonl in the trial log dir. Harbor bind-mounts the agent log dir so the file is already host-side, but a Pier --mounts-json run replaces the default log mounts, leaving the path dangling. The adapter now downloads runtime-events.jsonl alongside the cell output, skipping the download when the file already exists locally so Harbor's mounted path is untouched.
…er runners The pier copies of findTrialDir/readResultTrialName/readTrialException/ readCellOutput/providerTelemetryArtifactRefs/withProviderTelemetryArtifact had already drifted: readResultTrialName lost the exception_stats branch, so an errored trial whose name appears only there fell back to directory guessing. Parameterize the harbor originals with an InfraErrorCtor (defaulting to HarborInfraError, harbor call sites unchanged), export them, and delete the pier forks — restoring exception_stats parity and leaving only the documented 3-line utilities as local copies.
…raded outcome Harbor's authority order treats exception_info as how the agent phase ended, never as whether the trial was graded: only an ungraded budget exhaustion is a budget_exhausted outcome, and everything else falls through to the normal reward/cell reads. The pier runner instead hard-classified any non-budget exception as infra, discarding trials the verifier had graded (e.g. a non-zero Kimi CLI exit with reward=1). Align the flow, and when grading never produced a reward, surface the recorded trial exception as the root cause in the infra diagnostics, mirroring Harbor's readReward.
…s infra Every DeepSWE task's tests/test.sh traps EXIT with reward.txt=-1 when the verifier crashed before writing any reward file; grader.py documents that path as an infrastructure error and real rewards as binary 0/1, and pier's verifier parses the sentinel verbatim into verifier_result.rewards.reward. The runner accepted any finite number as a grade, so a crashed verifier was recorded as a scored failed sample instead of being excluded as infra.
…e-events fallback Harbor's hostTraceEventsPath never leaves traceEventsPath unset — it falls back to the runtime events — but the pier runner omitted the field when the combined trace was missing, and downstream trace analysis silently skips samples with an absent path. Align the runner, and have the shared maka adapter download trace-events.jsonl under Pier's custom mounts with the same already-host-side guard as runtime-events.jsonl.
…le model Pier runs a fixed 360s agent_setup phase (pier/trial/trial.py:176) that is scaled by the timeout multiplier like every other phase (pier/trial/execution.py:129-143); the lifecycle derivation omitted it, so at higher multipliers a slow setup could eat into the fixed setup/teardown grace. Also refresh the TaskRunner seam comment now that the Pier implementation exists alongside Harbor's.
Pier's --mounts-json replaces the default log mounts while capabilities.mounted stays true, so pier's own log download never runs and the CLI's /logs/agent/kimi-code.jsonl existed only in the container. Host-side _events(require_assistant=True) then raised 'did not contain an assistant message' — a real token-burning run misclassified as infra. The adapter now downloads the stream-json in run()'s finally (covering the AgentTimeoutError path too), skipping when the file is already host-side under Harbor's mount; it is _write_cell_output's only in-container input.
…ssion traces The pier runner's local two-branch trace resolution (combined trace else raw runtime events) skipped harbor's cell-mode branch that resolves the maka-storage session events via cell.runtimeRefs — the rich trace carrying tool_failed and provider_request_captured. On the maka host-cell arm that file exists host-side, so failure attribution downstream was silently emptied while the budget-recovery path (shared readTimedOutTrialArtifacts) resolved it correctly. Export hostTraceEventsPath and use it on the main path; delete the local simplification.
…er runner
grader.py documents the main reward as exactly 0 or 1 across all 113 tasks;
-1 is the crash sentinel (already infra) and any other value can only come
from corrupt or non-contract verifier output. The runner accepted any
non-negative finite number, so reward.json {"reward":0.5} scored as a PASS
through reward > 0. assertGradedPierReward now accepts exactly 0 or 1 and
classifies everything else as infra.
The serialization queue lived in each createPierTaskRunner closure, so two runner instances in one process (e.g. an A/B with two Kimi arms) each held their own lock and the second concurrent attempt on the shared fixed port was a guaranteed EADDRINUSE — a terminal infra under protectPassAtOne. The lock's natural owner is the shared host port: a module-level Map<port, queue> now serializes per port within the process (cross-process collisions remain the operator's scheduling concern, documented at the lock). Distinct ports and ephemeral binds stay concurrent.
…graded cells The controller's inherited semantics (isProviderInfraFailure exclusion vs structuredVerifierPassed override) predate this PR and apply to both runners identically; this parity test drives the controller with the pier runner's real output and the canonical harbor-shaped output for the same trial shape (graded reward 0 and 1, cell failed, errorClass=infra_failed) and asserts field-identical events, so neither side can drift unilaterally. Also type the shared InfraErrorCtor against a structural InfraErrorLike instead of the HarborInfraError class, since PierInfraError is deliberately not a subclass.
Replace readOptionalPierReward's throw-on-corrupt behavior with readPierGrade, which reduces both persisted mirrors of Pier's scoring authority (reward.json and result.json verifier_result.rewards.reward) to a single discriminated graded/ungraded/invalid value and never throws itself. - Enforce mirror consistency: when both mirrors carry a value they must be identical, else the grade is invalid (infra in the graded read path). - A reward.json that exists but has no finite numeric reward field is invalid, not a silent fall-through to the result mirror. - Split the two consumer contexts. The graded read path treats an invalid grade as infra. The budget gate treats invalid the same as ungraded: once the agent budget is spent, a crashed or corrupt verifier no longer overturns that fact, so the trial stays budget_exhausted (no retry, Pass@1 evidence preserved) rather than being retried as infra, and the invalid detail rides the exhaustion diagnostics.
Add providerProxyAdvertisedHost to the Kimi arm so operators can point the container's provider proxy at an explicit host. The default host.docker.internal is injected by Docker Desktop but absent on native Linux Docker, where pier 0.3.0's compose wires no extra_hosts/host-gateway, leaving the container's Squid unable to resolve the proxy. When set, the value is advertised in MAKA_PROVIDER_PROXY_URL (e.g. 172.17.0.1); unset preserves the current default. The Maka arm is unaffected and stays on loopback.
10 of 23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
createPierTaskRunner, aTaskRunnerimplementation over Datacurve's Pier (the DeepSWE Harbor fork), plus the provider-proxy port control the offline Kimi arm needs. This is the runner half of the DeepSWE A/B in #1343; the adapters were made Pier-loadable in #1344.createPierTaskRunnermirrorscreateHarborTaskRunnerbut drives Pier's CLI instead of a JobConfig file. Per task it spawnspier runwith the maka repo bind-mounted read-only at/opt/maka-agent, then maps the trial dir back toTaskRunOutput. Three env facts drove the design, each verified against the pilot run under~/.maka/eval/runs/deepswe-pilot-v1and pier 0.3.0 source:--ae KEY=VALUEcarries the non-secret agent identity/config (model, provider, pricing, cell timeout). Pier strips whitespace from--aevalues, so nothing whitespace-sensitive may ride here.MAKA_BACKEND(the adapterCliFlag'senv_fallbackreads onlyos.environ, so--ae MAKA_BACKEND=is silently ignored) andMAKA_SYSTEM_PROMPTbyte-exact — a trailing-newline prompt through--aewould be truncated byvalue.strip()and break the execution-identity hash round-trip on every task.0600--env-file(removed after the run) carries proxy-minted secrets so they stay off argv.Reward is read as one discriminated grading state (
readPierGrade) from both persisted mirrors of Pier's scoring authority — the DeepSWE verifier'sverifier/reward.jsoncross-checked against the trial'sverifier_result.rewards.reward; mirrors that disagree, a reward file without a numeric reward field, the crash sentinel, and non-binary values are allinvalid(infra on the graded read path). There is no Maka oracle verifier artifact, so the structuredharbor.verifieroutcome is constructed from the graded reward. Timeouts, trialexception_info, and budget exhaustion are classified so the fixed-prompt controller excludes infra from scoring and records budget exhaustion as a benchmark outcome. Cross-runner benchmark invariants are shared with the Harbor runner rather than copied: model-id normalization (modelIdForProvider), experiment-identity/pricing override rejection (assertNoExperimentIdentityOverrides), and timed-out trial artifact recovery (readTimedOutTrialArtifacts, so budget-exhausted samples keep their Pass@1 eligibility instead of shrinking the denominator asmissing_execution_identity).The Kimi arm runs the host provider auth proxy on a Squid-legal port and hands the container
MAKA_PROVIDER_PROXY_URL/TOKEN. To support that, the proxy now accepts an explicit fixed listen port (default stays ephemeral); Pier's egress Squid for offline tasks only permits destination ports 80/443 (acl Safe_ports port 80 443), and a privileged/busy bind fails with a clear message. Because the default port is fixed, concurrent Kimi attempts on one runner are serialized with a per-runner queue — only for fixed-port binds; an explicit ephemeral port (0) cannot collide and runs concurrently (per-attempt telemetry attribution preserved; a pool over 80+443 is deferred). In every arm the real provider key stays host-side — passed only as a file path (MAKA_HOST_API_KEY_FILE) or held by the proxy, which mints a scoped token; the container never receives it.On a watchdog timeout the runner terminates the detached pier process group with SIGTERM first — pier converts it to KeyboardInterrupt (
pier/cli/jobs.py) and runs its finally-based docker teardown — escalating to SIGKILL only after a 120s grace, so a timeout does not leak trial containers or an orphaned host cell. The watchdog budget models pier's complete trial lifecycle: 2×build + agent setup (a fixed multiplier-scaled 360s,pier/trial/trial.py:176) + agent + 2×verifier, because pier retries environment start and verification once each (pier/trial/execution.py:208,pier/trial/trial.py:333);environment.build_timeout_secnow rides task metadata. The Maka arm also covers keyless providers (registry authKindnone: ollama/lm-studio/localai) by running the host cell withMAKA_HOST_NO_AUTH, mirroring the Harbor runner's predicate, while backendfakekeeps the inert in-container cell. And because a Pier--mounts-jsonrun replaces the default log mounts whilecapabilities.mountedstays true (so pier's own log download never runs), both shared adapters compensate for host-side reachability of every in-container artifact they consume or promise: the maka adapter downloadsagent/runtime-events.jsonlandagent/trace-events.jsonlalongside the cell output, and the Kimi adapter downloads itskimi-code.jsonlstream — the cell output's only in-container input — inrun()'s finally so the AgentTimeoutError path hydrates too; each skips files already host-side under Harbor's bind mount.Trial classification follows Harbor's authority order exactly: only a budget exhaustion without an authoritative grade becomes a budget_exhausted outcome, every other trial exception falls through to the normal reward/cell reads — a graded trial scores on its actual reward (e.g. a non-zero Kimi CLI exit the verifier still passed), and missing artifacts become infra with the trial exception attached as the root cause. A negative reward is never a grade: DeepSWE's
tests/test.shtraps a verifier crash by writingreward.txt=-1(grader.py documents real rewards as binary 0/1), so on the graded read path the sentinel is infra instead of a scored failed sample. At the budget gate the precedence flips: the agent has already spent its budget, which is the authoritative fact, so aninvalidgrading authority (crashed/corrupt verifier, mismatched mirrors) is treated like ungraded — the sample stays budget_exhausted (no controller retry burning a second agent budget) with the grade detail carried in the diagnostics.traceEventsPathis always set, falling back to the runtime events like Harbor'shostTraceEventsPath, so downstream trace analysis never silently skips a Pier sample.Scope is the two arms the issue exercises (Maka host-side + offline container, and Kimi Code) plus the inert
fakebackend; the wide Harbor surface (GitHub Copilot, OpenCode, Codex) is not reimplemented. AdockerPlatformoption is deliberately absent: pier 0.3.0'sEnvironmentConfighas noextra_docker_compose/platform field, so it cannot be wired.fixed-prompt-controller.tsis untouched, using the canonicalTaskRunner/TaskRunInput/TaskRunOutputnames so a concurrent rename lands without conflict.Verification
npm test -w @maka/headless— 1265 tests, 1265 pass, 0 fail, 0 skipped (the pier venv is present locally, so even the pier harness-compat contract tests ran). New since the third round: a Kimi-adapter smoke lock for the Pier stream-json download (downloads under custom mounts, skips under Harbor's mount), graded-despite-exception and ungraded-exception-diagnostics tests for the fall-through classification, crash-sentinel tests for both reward sources, a trace-fallback test, a sharedfindTrialDirexception_stats parity test with pier-named diagnostics, and the watchdog contract updated to the full lifecycle including agent setup (2×1800 + 360 + 5400 + 2×1800 seconds + grace, ≥ 12,960,000 ms). New since the sixth round: mirror-agreement scoring, bidirectional mirror-mismatch → infra, reward.json-without-reward-field → infra, budget-gate precedence over the crash sentinel and over corrupt reward.json (→ budget_exhausted, not infra), and the Kimi advertised-host override.npm run lint,npm run format:check— clean.Live Pier + Docker fake trial (zero token, no API key), reusing the pilot's
maka-dasel-fake2recipe through the real runner and CLI, with a prompt whose bytes end in a newline to lock the--aestripping fix:Full chain passed (environment build, offline agent_setup, in-container fake cell, DeepSWE verifier), the cell's execution-identity hash matches the controller-side hash of the exact prompt bytes (confirming the fake backend still takes the in-container path with no
MAKA_HOST_*wiring), the returnedruntimeEventsPathexists host-side with the trial's real runtime events downloaded by the adapter, andtraceEventsPathresolves through the runtime-events fallback to an existing file.Benchmark-semantics inventory (Harbor mapping layer -> Pier disposition)
Two independent review arms converged on the same root cause class: the Pier mapping layer re-deriving benchmark semantics that
harbor-task-runneralready owns authoritatively. This inventory enumerates every benchmark semantic carried by the Harbor mapping layer and pins the Pier runner's disposition, as the convergence evidence against a third round of the same defect class.resolveNativeTrialTimeoutMsover a runner-supplied phase budget: Harbor supplies agent + oracle verifier outer seconds (byte-identical behavior); Pier supplies its complete lifecycle — 2×build + agent setup (fixed multiplier-scaled 360s,pier/trial/trial.py:176,pier/trial/execution.py:129-143) + agent + 2×verifier, because pier retries environment start and verification once each (pier/trial/execution.py:208,pier/trial/trial.py:333) — locked by a contract test against the DeepSWE task profilemodelIdForProviderassertNoExperimentIdentityOverrides(identical key set)assertNoProviderSecretsInAgentEnvreadTimedOutTrialArtifacts+harborTraceMode(same adapters write the sameagent/layout)gradedstate + cell output present, since Pier has no oracle verifier artifact; aninvalidgrading authority does not overturn the spent budget and is treated like ungraded at this gate); every other exception falls through to the normal reads — graded trials score, missing artifacts become infra with the trial exception attached (Harbor'sreadRewarddiagnostics parity)HarborVerifierOutcometype; Pier constructs a single-attempt outcome from its grading authority (verifier_result/reward.json) since no oracle JSON existsassertVerifierRewardAgreement, different artifact pair: Pier's two independently persisted mirrors of the grading authority (verifier/reward.jsonvsresult.jsonverifier_result.rewards.reward) must be identical;readPierGradeclassifies a disagreement (either direction) asinvalid→ infra on the graded read path, never a silently preferred scoreverifierFailureSummarystdout heuristicsproviderTokenSummary;PierTaskPricingis an alias ofHarborTaskPricingproviderProxyAuthMode/providerProxyUsageProtocolisBudgetExhaustedTrialException/isBudgetExhaustedErrormergeAgentEnvhostTraceEventsPath+ unit tests:traceEventsPathis always set — cell mode resolves theagent/maka-storage/sessions/<sid>/runs/<rid>/events.jsonlsession trace viacell.runtimeRefs(the rich trace withtool_failed/provider_request_captured; the earlier pier simplification skipped this branch and silently emptied downstream failure attribution on the maka host-cell arm), task-run mode the combinedagent/trace-events.jsonl, then the raw runtime events as last resortreadPierGrade: graded / ungraded / invalid) oververifier/reward.jsoncross-checked with the trialverifier_result.rewards.rewardmirror (Harbor readsreward.txt+ oracle outcome JSON; layouts differ); the DeepSWE contract is enforced as exactly binary 0/1 (grader.py) — -1 is thetest.shEXIT-trap crash sentinel, any other value (e.g. 0.5, whichreward > 0would otherwise score as a pass) can only be corrupt or non-contract verifier output, and a reward.json that exists without a numeric reward field is corrupt authority, allinvalid: infra on the graded read path, treated like ungraded at the budget gatefindTrialDir/readResultTrialName(parameterized by harness label +InfraErrorCtor): thereward_statsANDexception_statstrial-name hints plus the directory-name fallback are one implementation — the earlier pier fork had silently dropped the exception_stats branch, the exact drift class this inventory exists to prevent; parity locked by unit testfixed-prompt-controller, shared by both runners; the runner's obligation is byte-exact pass-through of cell output and identity (locked by the live-trial IDENTITY_MATCH evidence and the controller-level test)PierInfraErrorwith the rationale in code: the controller classifies by behavior, never identity, and diagnostics must name the failing harness; the shared trial-mapping and telemetry-attachment helpers take anInfraErrorCtorso each runner throws its own type through one implementationpier/cli/jobs.py); the runner invents no docker-scanning logicnone) provider predicateproviderRequiresSecret; a backendfakegate keeps the inert in-container path unchanged (locked by unit tests plus the live fake-trial evidence)--mounts-jsonreplaces pier's default log mounts whilecapabilities.mountedstays true, so pier's own log download never runs)_download_cell_outputfetchesruntime-events.jsonlandtrace-events.jsonlalongside the cell output; kimi_download_agent_logsfetcheskimi-code.jsonl(the cell output's only in-container input) inrun()'s finally so the AgentTimeoutError path hydrates too (adapter smoke tests for both adapters and both mount modes + live-trial file evidence)--env-file+ process env vs Harbor's job-configenv); locked by unit tests and the masked-config pilot evidencetail,errorText,isRecord,sanitize, optional readers)Scoring semantics inherited from the controller
One asymmetry surfaced in review is real but predates this PR and applies to both runners byte-identically: for a cell that crashed (
errorClass: 'infra_failed') the fixed-prompt controller excludes the sample viaisProviderInfraFailure(fixed-prompt-controller.ts:835,1196, scored=false, out of the denominator) when the graded reward is 0, but scores it as passed via thestructuredVerifierPassedoverride (fixed-prompt-controller.ts:820,873-878) when the graded reward is 1. This semantic was introduced onmainin the #1211/#1313 era and is consumed identically by the Harbor and Pier runners — this PR's invariant is cross-runner parity, not the controller's scoring philosophy, so it is deliberately left unchanged here; revisiting the asymmetry itself is a separate controller issue. A cross-runner parity test now drives the controller with the Pier runner's real output and the canonical Harbor-shaped output for the same trial shape (graded reward 0 and 1, cell failed,errorClass=infra_failed) and asserts field-identical controller events, so neither runner can drift from the other on this shape.Known risk
The pinned Kimi CLI is Node/undici and does not honor a Pier-injected
HTTP_PROXYby default, so whether the in-container model request actually reaches the host proxy through Squid can only be confirmed by the first real Kimi trial. If it does not, the Kimi arm's egress wiring needs a redesign. This is a recorded risk (also noted in code), not resolved here; the follow-up 30-task measurement in #1343 will surface it.Two further deployment notes for the first real trial's gate checklist:
pier/environments/agent_setup.py:144-214) adds noextra_hoststo its services, so inside the Squid containerhost.docker.internalonly resolves under Docker Desktop's magic DNS (macOS/Windows). On a plain Linux docker host the Kimi arm's first proxied request would fail (Squid 503 on name resolution). The current benchmark host is macOS; a Linux deployment must passproviderProxyAdvertisedHost(the escape hatch added for exactly this — e.g. the docker bridge address172.17.0.1) and verify reachability on the first trial.docker container prune) so stopped containers do not accumulate disk.