From 9c3f3596ee1f28134a03a696b200562ce5079086 Mon Sep 17 00:00:00 2001 From: Alec Wenzowski Date: Wed, 19 Aug 2026 22:32:19 +0000 Subject: [PATCH 1/2] fix(mcp): stop the budget gate justifying its floor with the number it falsified MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `32f68be` raised FLOOR 60000 -> 105000 on a re-measured worst observed success of 52747 ms and recorded that re-measurement in the header, but four passages of the pre-measurement text survived in the same file. The one that costs something is not a stale number. The `settings.json` `env` paragraph concluded, from ten logs that all opened at 30000ms, that the block does not reach the MCP client and therefore that no fix could live in this repository — and the gate emitted that same conclusion as its RUNTIME REMEDY, so the fix pointer CLOUD-122 requires pointed at the wrong place. It is false: the 14:28 container opened with `Starting connection with timeout of 120000ms` and succeeded at 52747 ms. Every one of those ten logs predated the connection that settles it. Why one container picks the declaration up and another does not is open, and the file now says so instead of answering it wrongly. Also corrected: the "1.8x headroom" and "~7x" margins, which derived from the superseded 16.65s and described the host default as tight when the later measurement shows it insufficient; and `inert` in the second `#MUTANT` rationale, the retracted word in the rationale for the row that exists because of the retraction. The recurrence gets a mechanism rather than a promise, adopting `timeout-check`'s existing `budget-arithmetic` class for the one budget its glob does not cover. The floor now carries its basis on the line it annotates and the gate parses it: FLOOR=105494 # budget: worst=52747ms x2 measured=2026-08-19 A floor moved without its basis fails, and so does a basis moved without the floor. 105494 rather than 105000 because 52747 x 2 is 105494 — the rounding that produced the older value is a small instance of the same drift, caught by the assertion on its first run. Nine bats rows cover both directions, an absent comment, two malformed ones, an unreadable budget file, and arithmetic refused even when the declared budget clears the floor. A third `#MUTANT` row drops the comparison. Closes CLOUD-730 --- mise-tasks/mcp-timeout-budget | 107 ++++++++++++++++++++++++++-------- tests/mcp-timeout-budget.bats | 89 +++++++++++++++++++++++++++- 2 files changed, 168 insertions(+), 28 deletions(-) diff --git a/mise-tasks/mcp-timeout-budget b/mise-tasks/mcp-timeout-budget index 56486e6d9..a71ec788c 100755 --- a/mise-tasks/mcp-timeout-budget +++ b/mise-tasks/mcp-timeout-budget @@ -49,13 +49,20 @@ # against 1.110 s idle), but a cold launch measured against the host default in a # container where the declaration had not reached the client. # -# WHY 120000 AND NOT 30000. The worst observed success is 16.65s, so the host -# default gave 1.8x headroom — and it came up short twice in one session. The -# cost function is deeply asymmetric: an over-long budget is paid only when the -# server is genuinely failing, once, at startup, whereas an under-set one costs -# every tool Serena owns for the rest of the session. 120000 is ~7x the worst -# observed success, which is margin for a cold container that is also running -# `mise install`, a cargo build and a first-time rust-analyzer index. +# WHY 120000 AND NOT 30000. Against the worst observed SUCCESS of 52747 ms the +# host default is not merely tight, it is BELOW the measurement: a connect that +# did succeed would have been killed by it, which is precisely the 07:06 row +# above. The cost function is deeply asymmetric — an over-long budget is paid +# only when the server is genuinely failing, once, at startup, whereas an +# under-set one costs every tool Serena owns for the rest of the session. 120000 +# is 2.3x the worst observed success, which is the margin a cold container needs +# while it is also running `mise install`, a cargo build and a first-time +# rust-analyzer index. +# +# THE 16.65s THIS SECTION USED TO CITE was the 2026-08-18 worst success, and the +# "1.8x headroom" and "~7x" margins derived from it are superseded by the +# re-measurement above (CLOUD-730). They were not merely stale: 1.8x described +# the host default as tight when the later measurement shows it insufficient. # # The FLOOR sits below the declared value so it can be tuned down on evidence # without editing this gate, while still refusing a return to the host default. @@ -63,21 +70,27 @@ # # It was 60000, derived from a worst observed success of 16650 ms. The 2026-08-19 # re-measurement above puts that at 52747 ms, which left the old floor at 1.14x — -# so it moves to 105000, twice the worst observed success, keeping the declared -# 120000 comfortably above it and the tuning slack intact. +# so it moves to 105494, exactly twice the worst observed success, keeping the +# declared 120000 above it and the tuning slack intact. # # NOT `.mcp.json`'s per-server `env`: that is passed to the CHILD, and this # budget governs the PARENT's wait. # -# AND NOT `.claude/settings.json`'s `env` BLOCK EITHER, which this header used to -# call "the one surface that reaches the CLI's own environment". Measured -# 2026-08-19 and false (CLOUD-700): the key is declared at 120000, `echo -# $MCP_TIMEOUT` inside the session prints 120000 — so the block does reach -# subprocesses — and every one of the ten recorded connections still opened with -# `Starting connection with timeout of 30000ms`. Both in-repo surfaces are -# therefore eliminated: the budget is read by the client before or independently -# of that layer, and only the environment the CLI is LAUNCHED with can set it, -# which is the remote environment's configuration and outside this repository. +# BUT `.claude/settings.json`'s `env` BLOCK IS THE SURFACE, and it does reach the +# client — just not on every container, which is the whole reason this gate reads +# the log instead of the file. +# +# CORRECTED 2026-08-19 (CLOUD-730). This header previously concluded, from ten +# logs that all opened at `30000ms`, that the block "does not reach the MCP +# client", that BOTH in-repo surfaces were eliminated, and therefore that only +# the environment the CLI is launched with could set the budget — i.e. that no +# fix could live in this repository. That conclusion is FALSE, and the sample is +# why: every one of those ten logs predated the connection that settles it. The +# 14:28 container opened with `Starting connection with timeout of 120000ms`. +# +# WHAT IS STILL OPEN is why one container picks the declaration up and another +# does not. That question is live and this file does not answer it. Nothing here +# should be read as retiring the settings surface. # # THAT IS WHY THIS GATE ALSO ASSERTS THE EFFECT. Checking the declaration alone # measures what the repo says while the failure is entirely in what the host @@ -95,16 +108,60 @@ # host default 30000 is a declared value, so a gate that only checks presence # passes the exact state CLOUD-668 was filed about. #MUTANT presence-is-not-a-budget|s/\[ "\$declared" -ge "\$FLOOR" \]/[ -n "$declared" ]/|a value below the floor is refused, and both numbers are named -# The second mutation drops the effect half, leaving the declaration check that -# passed for ten sessions while the budget was inert. Only a row that reads the -# LOG can catch it. +# The second mutation drops the effect half, leaving the declaration check alone +# — which passes on any container where the client used the host default while +# this file declared 120000, the exact 07:06 case above. Only a row that reads +# the LOG can catch it. #MUTANT declaration-without-effect|s/\[ "\$observed" -lt "\$declared" \]/false/|an observed budget below the declared one is refused +# The third mutation drops the floor's own arithmetic, so a floor may be moved +# without moving the measurement that justifies it — the drift that let a +# superseded "16.65s" go on justifying a number derived from 52747 ms. Only a row +# that declares a basis disagreeing with its floor can catch it. +#MUTANT floor-without-basis|s/\[ "\$FLOOR" -ne \$((budget_worst \* budget_multiplier)) \]/false/|a floor disagreeing with its declared basis is refused #PIN-OK: jq set -uo pipefail -# The measured floor. Raising the declared value is free; lowering this needs a -# new row in the measurement table above. -FLOOR=105000 +# THE FLOOR AND THE MEASUREMENT THAT JUSTIFIES IT, on one line, in the form +# `timeout-check` already uses for workflow jobs (CLOUD-266): +# +# FLOOR= # budget: worst=ms x measured=YYYY-MM-DD +# +# and it is PARSED below rather than merely readable. A floor moved without its +# basis fails, and so does a basis moved without the floor — which is exactly how +# a "16.65s" justification outlived the number it justified (CLOUD-730). Raising +# the declared value in settings is free; moving this needs a new row in the +# measurement table above and the arithmetic to match it. +# +# 105494 rather than the 105000 that first landed: 52747 x 2 is 105494, and the +# rounding that produced the older value is a small instance of the same drift +# this assertion exists to refuse. +FLOOR=105494 # budget: worst=52747ms x2 measured=2026-08-19 + +# Parsed from the file the assignment above lives in, so a fixture can exercise +# every direction of the check by declaring its own line. `timeout-check` is a +# pure function of the workflows it reads; this is a pure function of one line of +# this file. +budget_file="${MCP_TIMEOUT_BUDGET:-${BASH_SOURCE[0]}}" +if [ ! -r "$budget_file" ]; then + echo "::error:: mcp-timeout-budget: cannot read $budget_file to check the floor's own budget comment" >&2 + exit 2 +fi +budget_line=$(grep -m1 -E '^FLOOR=' "$budget_file") +budget_re='^FLOOR=([0-9]+)[[:space:]]#[[:space:]]budget:[[:space:]]worst=([0-9]+)ms[[:space:]]x([0-9]+)[[:space:]]measured=[0-9]{4}-[0-9]{2}-[0-9]{2}$' +if [[ ! $budget_line =~ $budget_re ]]; then + echo "::error:: mcp-timeout-budget: the floor carries no parsable budget comment — a limit with no recorded measurement is the boilerplate CLOUD-266 refuses" >&2 + echo " want FLOOR= # budget: worst=ms x measured=YYYY-MM-DD" >&2 + exit 1 +fi +FLOOR="${BASH_REMATCH[1]}" +budget_worst="${BASH_REMATCH[2]}" +budget_multiplier="${BASH_REMATCH[3]}" +if [ "$FLOOR" -ne $((budget_worst * budget_multiplier)) ]; then + echo "::error:: mcp-timeout-budget: the floor disagrees with the basis it declares" >&2 + echo " floor $FLOOR" >&2 + echo " basis $budget_worst x$budget_multiplier = $((budget_worst * budget_multiplier))" >&2 + exit 1 +fi settings="${MCP_TIMEOUT_SETTINGS:-.claude/settings.json}" logroot="${MCP_TIMEOUT_LOGS:-}" @@ -210,7 +267,7 @@ observed_budget_holds() { echo "::error:: mcp-timeout-budget: the host ignored the declared MCP startup budget" >&2 echo " $server observed $observed" >&2 echo " $settings env.MCP_TIMEOUT $declared" >&2 - echo " A declared budget the client does not use is not a budget. The settings file's env block reaches subprocesses and not the MCP client (CLOUD-700), so this has to be set in the environment the CLI is launched with — the remote environment's own configuration, not this repository." >&2 + echo " A declared budget the client does not use is not a budget. Why this container used a different one is OPEN (CLOUD-730): the declaration in this settings file demonstrably does reach the client on other containers, so do not 'fix' this by moving the key out of the repository. Serena had the observed budget, not the declared one, to attach in." >&2 return 1 fi done <<<"$servers" diff --git a/tests/mcp-timeout-budget.bats b/tests/mcp-timeout-budget.bats index d8f54b073..c86bbd701 100644 --- a/tests/mcp-timeout-budget.bats +++ b/tests/mcp-timeout-budget.bats @@ -45,17 +45,17 @@ log_connection() { # log_connection [stamp] run "$GATE" --settings "$S" [ "$status" -eq 1 ] [[ "$output" == *"30000"* ]] - [[ "$output" == *"105000"* ]] + [[ "$output" == *"105494"* ]] } @test "exactly the floor passes — the bound is inclusive" { - declare_timeout 105000 + declare_timeout 105494 run "$GATE" --settings "$S" [ "$status" -eq 0 ] } @test "one millisecond under the floor is refused" { - declare_timeout 104999 + declare_timeout 105493 run "$GATE" --settings "$S" [ "$status" -eq 1 ] } @@ -194,3 +194,86 @@ log_connection() { # log_connection [stamp] run "$GATE" --settings "$S" --logs [ "$status" -eq 2 ] } + +# --- the floor carries its own arithmetic (CLOUD-730) ------------------------ +# +# `timeout-check` refuses a workflow budget whose declared minutes disagree with +# its own stated p95 x multiplier. These rows are that failure class for the one +# budget its glob does not cover, and they exist because the drift already +# happened: the floor moved 60000 -> 105494 on a re-measured worst success of +# 52747 ms while the header went on justifying it with the superseded 16.65s. +# +# The gate parses the FLOOR line out of `$MCP_TIMEOUT_BUDGET`, defaulting to its +# own file, so a fixture can declare every direction. The floor the parse yields +# is the floor enforced — the number and its basis cannot be varied apart. + +budget_file() { # budget_file + local f="$BATS_TEST_TMPDIR/budget.sh" + printf '%s\n' "$1" >"$f" + printf '%s' "$f" +} + +@test "a floor equal to its declared basis passes" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=105494 # budget: worst=52747ms x2 measured=2026-08-19')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 0 ] +} + +@test "a floor raised without moving its basis is refused, and both numbers are named" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=200000 # budget: worst=52747ms x2 measured=2026-08-19')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 1 ] + [[ "$output" == *"200000"* ]] + [[ "$output" == *"105494"* ]] + [[ "$output" == *"disagrees with the basis"* ]] +} + +@test "a basis moved without the floor is refused — drift in either direction" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=105494 # budget: worst=90000ms x2 measured=2026-08-19')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 1 ] + [[ "$output" == *"disagrees with the basis"* ]] +} + +@test "a floor with no budget comment is refused — a limit with no measurement" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=105494')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 1 ] + [[ "$output" == *"no parsable budget comment"* ]] +} + +@test "a malformed budget comment is refused rather than parsed loosely" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=105494 # budget: about twice the worst one')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 1 ] + [[ "$output" == *"no parsable budget comment"* ]] +} + +@test "a budget comment with no measurement date is refused" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=105494 # budget: worst=52747ms x2')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 1 ] + [[ "$output" == *"no parsable budget comment"* ]] +} + +@test "an unreadable budget file is exit 2, never a silent pass" { + declare_timeout 120000 + MCP_TIMEOUT_BUDGET="$BATS_TEST_TMPDIR/absent-budget.sh" \ + run "$GATE" --settings "$S" + [ "$status" -eq 2 ] +} + +# The arithmetic is checked BEFORE the declaration is compared, so a repo whose +# floor has drifted cannot report a green budget on the strength of it. +@test "the arithmetic is refused even when the declared budget clears the floor" { + declare_timeout 999999 + MCP_TIMEOUT_BUDGET="$(budget_file 'FLOOR=1 # budget: worst=52747ms x2 measured=2026-08-19')" \ + run "$GATE" --settings "$S" + [ "$status" -eq 1 ] +} From 4b0da083be5c17e0e1c7ba52442655058d85efe1 Mon Sep 17 00:00:00 2001 From: Alec Wenzowski Date: Wed, 19 Aug 2026 22:34:44 +0000 Subject: [PATCH 2/2] fix(mcp): name an existing case in the floor-arithmetic mutation `mise run mutant` refused the new row with `names-no-case`: its description is matched against a bats test name, and mine described the mutation rather than naming the row that catches it. Aligned to the case that actually kills it. Refs: CLOUD-730 --- mise-tasks/mcp-timeout-budget | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mise-tasks/mcp-timeout-budget b/mise-tasks/mcp-timeout-budget index a71ec788c..0ed3bb6f9 100755 --- a/mise-tasks/mcp-timeout-budget +++ b/mise-tasks/mcp-timeout-budget @@ -117,7 +117,7 @@ # without moving the measurement that justifies it — the drift that let a # superseded "16.65s" go on justifying a number derived from 52747 ms. Only a row # that declares a basis disagreeing with its floor can catch it. -#MUTANT floor-without-basis|s/\[ "\$FLOOR" -ne \$((budget_worst \* budget_multiplier)) \]/false/|a floor disagreeing with its declared basis is refused +#MUTANT floor-without-basis|s/\[ "\$FLOOR" -ne \$((budget_worst \* budget_multiplier)) \]/false/|a floor raised without moving its basis is refused, and both numbers are named #PIN-OK: jq set -uo pipefail