Skip to content

fix(mcp): harden /mcp from the audit (address budget, Origin, tool contracts) - #1225

Merged
mforce merged 28 commits into
mainfrom
fix/mcp-audit-hardening
Oct 11, 2026
Merged

mforce merged 28 commits into
mainfrom
fix/mcp-audit-hardening

Conversation

@mforce

@mforce mforce commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Part of #789

Docs-impact: none — the glossary's "connected app" entry changed only its idempotency sentence, from "forgets the key after 48 hours" to "kept for at least 48 hours, after which it may be forgotten", matching the tool description. The in-app definition (help.glossaryConnectedAppDef, en/es/tl) is left alone on the maintainer's decision, because the Help precache has no room.

Fixes seven findings from the two read-only audits of the MCP server at 04dabbb, then the findings of three Codex review rounds on this PR (R1 to R11). Each finding has its own commit. The behaviour changes are A (refused bearers spend a per-address budget) and D (403 for a foreign or malformed Origin). B and C change tool description text only. E and J tighten guards. F adds one sort key.

Two changes reach operators and clients. A client address that keeps presenting refused bearers is cut off for the rest of the minute, valid tokens included. A browser page on another origin can no longer reach /mcp. No configuration key was added, so the sim harness and the AppHost need no change (#370, #565).

A. Forged bearers no longer get a fresh budget each (COD-02, F-01, review R1)

Finding. oauth-api partitions on a hash of any text after Bearer , before authentication. From one address, a new forged token on every request never reached a limit, and each one cost an OpenIddict token lookup.

Fix. oauth-api endpoints also count refused bearers per client address, in the shared IFixedWindowCounter (#543/#544), never in the process (#271). UseOAuthApiAddressLimits runs ahead of UseRateLimiter and authentication:

  • It reads the address's count with the new CountAsync, and answers a JSON-RPC 429 with Retry-After once the count reaches 300 in the minute.
  • After the response, a presented bearer answered 401 spends one.
  • A valid call, a scope 403, a per-token 429 and a tokenless discovery probe spend nothing, so farms behind one assistant's egress address no longer share a call budget.

300 a minute caps forged-token lookups at five a second per address and leaves room for revoked connections to retry before their people reconnect them. The per-token budget is unchanged.

Refusals reach that budget only when they finish, so the same middleware also caps bearer authentications in flight per address at 32, with no queue (review R6). It is a PartitionedRateLimiter concurrency limiter, per process on purpose, because in-flight work is each instance's own. The code and docs/decisions/271-single-serving-instance.md record that classification. UseOAuthApiInFlightRelease, right after CredentialEpochMiddleware, gives the permit back once the credential gates decide, so a valid call's body and tool work hold none (review R9). Nothing authenticates twice. The outer using releases it on a refusal, an exception or a cancellation.

IFixedWindowCounter.CountAsync reads without incrementing. Redis does it in one script on its own clock, and the in-process counter under its lock. The resilient decorator returns the larger of the Redis and fallback counts, because Redis may answer reads while refusing increments (review R7). The larger, not the sum, because an increment Redis applied before its reply timed out lands in both (review R10). It falls back and alarms when the read throws.

The first version of this PR counted every call per address, at ten tokens' worth. The review showed that one busy connection could exhaust it for every farm on a shared address, so it is gone.

Costs. Once an address has spent its failure budget, its valid tokens are refused too until the minute rolls over, because nothing tells them apart before authentication. Check-then-record lets requests that passed the check before earlier refusals landed overshoot, by no fixed number. The cap bounds the authentication work running at once, not that total (review R11). The ponytail: comment on FailedAuthenticationBudget records that ceiling.

Tests.

  • ForgedBearers_FromOneAddress_RunOutOfTheFailureBudget: 300 forged bearers get 401, and the 301st gets 429. This and the two count-inspection tests run on a counter over a stationary FakeTimeProvider (review R8).
  • SpentFailureBudget_RecoversWhenTheWindowRollsOver: a spent address gets 429, and after the clock advances one window it gets 401.
  • BearersInFlight_FromOneAddress_AreCapped_AndAnotherAddressIsNot: authentication is held open. 32 bearers from one address are in flight, the 33rd gets 429, and another address still enters authentication. After the gate opens, a further bearer from the first address gets 401.
  • ValidTokenCalls_SpendNoFailureBudget: five valid calls leave the count at 0, and one forged call makes it 1.
  • PerTokenRefusals_SpendNoFailureBudget: 200, 200, 429, 429 with the count still 0.
  • SpentFailureBudget_InTheSharedStore_RefusesBeforeAuthentication: only the shared store reports the budget spent, and a valid token gets 429 with that store's Retry-After.
  • AuthenticatedWorkInFlight_DoesNotHoldAuthenticationPermits: 32 authenticated requests from one address hold their tool work open, and a fresh authentication from that address gets 200.
  • CountAsync contract tests run in process, against a real Redis (two counters on one namespace) and through the resilient fallback. That includes a primary that reads but refuses increments, and one that applies increments and then times out.

B. grades: [] is documented (F-02)

Finding. An empty grades list removes a draft's grade lines. The parameter description said only "Sellable eggs per grade".

Fix. The description says that leaving it out keeps the draft's lines and that an empty list removes them.

Test. OmittedGrades_KeepTheDraftsLines_AndAnEmptyListRemovesThem records Large 90, records again without grades (the line stays), then records with [] (no lines).

C. The idempotency key states its retention (COD-04, review R4)

Finding. The idempotencyKey description promised replay with no time limit. IdempotencyRecordPurgeSweep deletes completed records some time after 48 hours.

Fix. Description text, plus the matching glossary sentence. A successful result is kept for at least 48 hours. After that the key may be purged and a retry may record again, so the caller checks the current entry first with cluckwork_list_daily_entries, or asks a person when the connection cannot read. Retention is unchanged.

D. /mcp refuses a foreign Origin (COD-01, F-10, review R2 and R3)

Finding. Nothing validated Origin. MCP 2025-11-25 and 2026-07-28 both require it. The SDK leaves this check to the app: its maintainers declined an SDK Origin filter (modelcontextprotocol/csharp-sdk#1819) in favour of ASP.NET Core's AllowedHosts and CORS (modelcontextprotocol/csharp-sdk#1554).

Fix. UseMcpOriginCheck runs ahead of rate limiting and authentication. It matches the /mcp path as routing does, ignoring case with one optional trailing slash, whatever endpoint routing selected. DELETE, PUT and OPTIONS, which select routing's own 405, are covered (review R3). A request without Origin passes, because native clients send none. Otherwise the request must carry exactly one value with no path, query, fragment or credentials, so null fails. That value's scheme, ASCII host (IdnHost) and effective port must equal the configured OAuth:Issuer's. The request's Host plays no part. CluckworkIdentityRegistration now carries the issuer Uri instead of a bool.

Tests.

  • ForeignOrigin_Gets403_BeforeAuthentication_OnEveryMethod, once per revision: a foreign Origin gets 403 on an unauthenticated POST, on GET, on DELETE and on POST to /MCP/. Without Origin, POST gets 401 and GET and DELETE get 405.
  • ForeignOrMalformedOrigin_Gets403_AndTheIssuersOriginConnects, once per revision, with the SDK client.
  • RequiredOrigin_ComparesSchemeAsciiHostAndEffectivePort, 18 rows. They include a Unicode issuer with a punycode header, :443 against an implicit port, duplicate values, a duplicate plus an empty value, and null.

E. The free-text guard walks write replies (COD-05, F-05)

Finding. RealTree_ReadToolRepliesReturnOnlyReviewedText started from read tools only, so a note added to RecordedDailyEntry passed green.

Fix. The test, now RealTree_ToolRepliesReturnOnlyReviewedText, walks every tool method, as the money guard does. RecordedDailyEntry.FlockName joins the reviewed list.

F. Stock pages have a total order (COD-03, review R5)

Finding. GetStockByGradeAsync sorted by SortOrder then GradeName. Grade names are unique per farm, not per account, so two farms' "Large" tied, and PostgreSQL returns tied rows in no promised order.

Fix. .ThenBy(r => r.EggGradeId), in EggLotRepository.InStockOrder.

Tests.

  • StockOrder_BreaksANameAndSortOrderTieById hands the ordering two tied rows in descending id order.
  • ReadStock_PagesTiedGradesOfTwoFarms_EachOnce checks through the tool that each tied row is paged once with limit: 1. It makes no claim about read order.

J. Read tools set OpenWorld = false (F-07)

Finding. The nine read tools left openWorldHint at the spec default true.

Fix. All nine set OpenWorld = false. McpToolSurface.GateFindings fails a read tool without it, and the OpenWorldReadTool fixture holds the finding.

Mutations

Each mutation was applied to committed code, its tests run, and the file restored. Every one went red except the first F mutation, noted in its row.

Mutation Red tests
A never record a refusal ForgedBearers_FromOneAddress_RunOutOfTheFailureBudget
A record every response that carried a bearer ValidTokenCalls_SpendNoFailureBudget, PerTokenRefusals_SpendNoFailureBudget
A never refuse before authentication ForgedBearers_…, SpentFailureBudget_InTheSharedStore_RefusesBeforeAuthentication
A Redis CountAsync uses INCR instead of GET CountAsync_reads_the_shared_count_without_incrementing
A in-process CountAsync reads one too many CountAsync_ReadsWithoutIncrementing_AndResetsWithTheWindow
A resilient CountAsync rethrows instead of falling back CountAsync_WhenRedisThrows_ReadsTheFallbackAndAlarms
A resilient CountAsync ignores the fallback when Redis reads CountAsync_WhenRedisReadsButRefusesIncrements_AddsTheFallbackCount
A resilient CountAsync sums instead of taking the larger CountAsync_AfterIncrementsAppliedThenTimedOut_DoesNotDoubleCount
A the in-flight release step does nothing AuthenticatedWorkInFlight_DoesNotHoldAuthenticationPermits
A the release step removed from Program.cs the same test, and CredentialEpochMiddlewareOrderTests
A the permit released before authentication BearersInFlight_…
A the in-flight cap never refuses BearersInFlight_…. The 33rd waited on the gate until the 30 s bound.
A the in-flight lease is never released BearersInFlight_…. The request after the gate opened got 429.
A one in-flight partition for every address BearersInFlight_…. The other address waited on the gate.
B if (grades is not null) becomes if (grades is { Count: > 0 }) OmittedGrades_KeepTheDraftsLines_AndAnEmptyListRemovesThem. (Large, 90) stayed after [].
D GET/HEAD mapping without the origin metadata (metadata version) ForeignOrigin_Gets403_BeforeAuthentication_OnEveryMethod, both revisions
D middleware removed, so only the endpoint answers (metadata version) the same test, both revisions
D DELETE skips the path check the same test, both revisions
D trailing slash not matched the same test, both revisions
D first of several values accepted the duplicate and duplicate-plus-empty rows
D Host instead of IdnHost the Unicode-issuer row
D path, query, fragment and credentials allowed five rows
D (first version) the endpoint check always dispatches ForeignOrMalformedOrigin_Gets403_AndTheIssuersOriginConnects, both revisions
E RecordedDailyEntry gains string? Note RealTree_ToolRepliesReturnOnlyReviewedText
F .ThenBy(r => r.EggGradeId) removed StockOrder_BreaksANameAndSortOrderTieById, [e2b8…, 1bbd…] against [1bbd…, e2b8…]. The first, integration-only test stayed green under this mutation, which is why the ordering is now tested directly.
J OpenWorld = false removed from cluckwork_read_stock RealTree_EveryToolCarriesARolePolicyAndAScopePolicy

C changes description text only and has no test.

Live MCP client run

The first run, at 3b2c6703, is in the first live-run comment. The rerun at af0b7397, after the review fixes, used real consent with the password step-up and is in a later comment. In summary:

  • From one address, 20 valid calls got 200 and spent nothing. A burst of 330 forged bearers then got 301 401s and 29 429s. The extra 401 is the in-flight overshoot.
  • After the burst that address was refused, valid token included, while another address's valid token got 200.
  • A foreign Origin got 403 on an unauthenticated POST and on GET for both revisions, and null and a path also got 403. Without Origin, the same requests got 401 and 405.
  • openWorldHint: false on all nine read tools.
  • grades left out kept two lines, and [] removed them.
  • The new idempotencyKey text is served as written.
  • read_stock paged five grades once each.
  • The SDK client called a tool on both revisions.

Out of scope

F-03 (binding errors), F-04 (flock-name case), F-06 (superseded tokens), F-08 (discovery behind an untrusted proxy), F-09 (per-app keys) and COD-06 (the SDK's parse-error code) are untouched.

Local verification

At 644bca77, dotnet test tests/Cluckwork.Api.IntegrationTests filtered to Mcp, RateLimit, OAuth, SharedState, DistributedIp and the middleware order guard passed: 445 tests, 0 failed.

mforce and others added 9 commits October 11, 2026 06:41
The per-token key hashes any text after Bearer, so each forged token got a fresh 300/min window. A global limiter on the shared counter now gives each address ten tokens' worth on oauth-api endpoints, before OpenIddict looks the token up.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both MCP revisions require Origin validation against DNS rebinding. An absent Origin passes; any other value must equal the OAuth:Issuer origin.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An empty grades list removes a draft's grade lines; only omission was documented. A completed idempotency record is purged after 48 hours, after which a retry records again.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The #792 free-text walk visited read tools only, so a note on a write reply passed green.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
openWorldHint defaults to true, so clients treated nine database reads as external calls. The gate walk now fails a read tool without OpenWorld = false.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grade names are unique per farm, not per account, so two farms' same-named grades at one sort order had no total order across read_stock pages.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r is the id's

The first version passed with the tie-breaker removed: the grade query can read through the (account, farm, name) index, which orders by the random farm id. The farms are now ordered against the grade ids.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Live MCP client run at 3b2c6703, on an isolated stack (cwmcpfix, app on :8097) built from this branch. A TLS proxy on https://localhost:8496 fronts it, OAuth:Issuer is that origin, AllowedHosts=localhost and TRUSTED_PROXY_CIDR=172.16.0.0/12. The shared cluckwork-sim stack was not touched.

Consent. Two Owner tokens went through dynamic registration, sign-in, /connect with the mandatory password step-up, and the code exchange. The first is farm:read and the second holds both scopes. "Allow" with no password stayed on the consent screen. The read token lists the nine read tools, and the write token adds cluckwork_record_daily_entry.

J. openWorldHint. A raw tools/list returns readOnlyHint: true, openWorldHint: false for all nine read tools.

D. Origin.

2025-11-25 foreign https://untrusted.example -> 403
2025-11-25 malformed null -> 403
2026-07-28 foreign https://untrusted.example -> 403
2026-07-28 malformed null -> 403
2025-11-25 issuer origin https://localhost:8496 -> 200
2025-11-25 no Origin -> 200

The SDK client (probe.cs, which sends no Origin) negotiated 2026-07-28 and 2025-11-25 and called cluckwork_read_stock on each.

A. Forged-bearer burst. The app was hit directly, with X-Forwarded-For from the trusted Docker gateway, so that two client addresses could be compared. 3,050 POSTs, each carrying a different forged bearer from 203.0.113.50, ran inside one minute:

   3000 401
     50 429
forged bearer again from 203.0.113.50 -> {"jsonrpc":"2.0","id":null,"error":{"code":-32000,"message":"Too many requests. Try again later."}} 429
real token from 203.0.113.51 -> 200
real token through the TLS proxy (127.0.0.1) -> 200

The budget is 300 × 10 = 3,000 per address per minute, and the 3,001st request was the first 429.

B. grades: []. These calls recorded Sim Z Flock Catalog 001 on 2026-10-10 three times, each with a new key:

grades Large 90, Medium 4 -> [{"grade":"Large","quantity":90},{"grade":"Medium","quantity":4}]
grades left out          -> [{"grade":"Large","quantity":90},{"grade":"Medium","quantity":4}]
grades: []               -> []

B and C. Descriptions as served.

grades: Sellable eggs per grade, by grade name or id. Leave it out to keep the draft's existing grade lines; an empty list removes them.
idempotencyKey: A key you choose for this recording. For 48 hours after the first call completes, resending the same call with the same key returns the first result instead of recording again, and reusing it with different values is refused. After that the key is forgotten and a retry records again, so read the current entry with cluckwork_list_daily_entries before retrying an older call.

F. read_stock paging. With limit: 1, the five pages returned Small, Medium, Large, Cracked and Dirty once each, and the last page said complete: true, nextOffset: null. The fixture has one farm, so it has no tied grades. The tie case is covered by ReadStock_PagesTiedGradesOfTwoFarms_InIdOrder only.

The attached frame is the consent screen at 1280 px, light theme, 1:1.

Consent screen at 1280 px, light theme, with the password step-up

mforce and others added 7 commits October 11, 2026 06:52
Expiry depends on a later purge sweep, so a key is not forgotten at exactly 48 hours. A write-only connection checks the current entry through a person.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostgreSQL returns tied rows in no promised order, so the integration case cannot prove the id tie-breaker. The ordering moves to InStockOrder, and a test hands it tied rows in descending id order. The integration case now checks only that each tied row is paged once.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The all-call address budget let one busy connection, or plain valid traffic, throttle every farm behind a shared egress address. Now a presented bearer answered 401 spends the address's failure budget in the shared counter, and an address over budget is refused before authentication. Valid calls, per-token 429s and tokenless probes spend nothing. IFixedWindowCounter gains CountAsync, a read that leaves the count alone, on Redis, in process and through the resilient fallback.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check ran inside the authenticated POST endpoint, so an unauthenticated POST got the 401 challenge and GET or HEAD got 405 whatever its Origin. Both /mcp mappings now carry the issuer's origin as metadata, and a middleware ahead of rate limiting and authentication answers a foreign Origin with 403.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The flattened-string comparison let ["https://localhost", ""] through, refused a punycode Origin against a Unicode issuer, and refused an explicit default port. A present Origin must now be one origin-only value, compared by scheme, IdnHost and Port.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Reply to the Codex review at 7dd8dac3 (R1 to R5). Each finding has its own commit, pushed as af0b7397.

R1 (P2). Fixed in 63be9e77, following the maintainer's decision. The all-call address budget is gone. oauth-api endpoints now count refused bearers per client address in the shared IFixedWindowCounter. A middleware ahead of UseRateLimiter and authentication reads the address's count and answers 429 once it reaches 300 a minute. After the response, only a presented bearer answered 401 spends one. A valid call, a scope 403, a per-token 429 and a tokenless discovery probe spend nothing. IFixedWindowCounter gains CountAsync, which reads without incrementing. Redis does it in one script on its own clock, and the in-process counter does it under its lock. The resilient decorator falls back and alarms exactly as IncrementAsync does. 300 a minute caps forged-token lookups at five a second per address and leaves room for revoked connections behind one egress address to retry. It is a constant, so no new config key and no #370/#565 wiring. Tests:

  • ForgedBearers_FromOneAddress_RunOutOfTheFailureBudget: 300 forged bearers get 401, and the 301st gets 429.
  • ValidTokenCalls_SpendNoFailureBudget: five valid calls leave the count at 0, and one forged call makes it 1.
  • PerTokenRefusals_SpendNoFailureBudget: 200, 200, 429, 429 with the count still 0.
  • SpentFailureBudget_InTheSharedStore_RefusesBeforeAuthentication: only the shared store reports the budget spent, and a valid token gets 429 with that store's Retry-After.
  • CountAsync contract tests for in-process, real Redis (two counters on one namespace) and the resilient fallback.

Two costs are known and recorded. Once an address has spent its failure budget, valid tokens from it are refused too until the minute rolls over, because nothing can tell them apart before authentication. Check-then-record also lets the requests already in flight when the count crosses the limit through once per window. The ponytail: comment on FailedAuthenticationBudget records that ceiling.

R2 (P3). Fixed in fcd318f8. A present Origin must be exactly one value with no path, query, fragment or credentials, and null fails. It is compared by scheme, IdnHost (punycode) and effective Port. RequiredOrigin_ComparesSchemeAsciiHostAndEffectivePort covers a Unicode issuer with a punycode header, https://localhost:443, duplicate values, a duplicate plus an empty value, a lone empty value, null, a trailing slash, a path, a query, a fragment, user info, another scheme, another port and another host.

R3 (P3). Fixed in e3312e35. Both /mcp mappings carry the issuer's origin as metadata, and UseMcpOriginCheck runs ahead of rate limiting and authentication. ForeignOrigin_Gets403_BeforeAuthentication_OnEveryMethod runs once per revision. A foreign Origin gets 403 on an unauthenticated POST and on GET. Without Origin, the same requests still get 401 and 405.

R4 (P3). Fixed in 4cc5a170. The description now says a successful result is kept for at least 48 hours, after which the key may be purged and a retry may record again. It tells the caller to check the current entry first, or to ask a person when the connection cannot read.

R5 (P3). Fixed in 064e67ca. The ordering moved to EggLotRepository.InStockOrder. StockOrder_BreaksANameAndSortOrderTieById hands it two tied rows in descending id order. With .ThenBy(r => r.EggGradeId) removed it fails, reading [e2b8…, 1bbd…] against the expected [1bbd…, e2b8…]. The integration case is now ReadStock_PagesTiedGradesOfTwoFarms_EachOnce and claims only that each tied row is paged once. The insertion-order claim is gone.

Mutations. Each mutation below was applied to the committed code, its tests run, and the file restored. All 11 went red.

Finding Mutation Red tests
R1 never record a refusal ForgedBearers_FromOneAddress_RunOutOfTheFailureBudget
R1 record every response that carried a bearer ValidTokenCalls_SpendNoFailureBudget, PerTokenRefusals_SpendNoFailureBudget
R1 never refuse before authentication ForgedBearers_…, SpentFailureBudget_InTheSharedStore_RefusesBeforeAuthentication
R1 Redis CountAsync uses INCR instead of GET CountAsync_reads_the_shared_count_without_incrementing
R1 in-process CountAsync reads one too many CountAsync_ReadsWithoutIncrementing_AndResetsWithTheWindow
R1 resilient CountAsync rethrows instead of falling back CountAsync_WhenRedisThrows_ReadsTheFallbackAndAlarms
R3 GET/HEAD mapping without the origin metadata ForeignOrigin_Gets403_BeforeAuthentication_OnEveryMethod, both revisions
R3 middleware removed, so only the endpoint answers the same test, both revisions
R2 first of several values accepted the duplicate and duplicate-plus-empty rows
R2 Host instead of IdnHost the Unicode issuer with a punycode header
R2 path, query, fragment and credentials allowed five rows (trailing slash, path, query, fragment, user info)
R5 .ThenBy(r => r.EggGradeId) removed StockOrder_BreaksANameAndSortOrderTieById

The targeted suite at af0b7397 passed: 437 tests across Mcp, RateLimit, OAuth, SharedState and DistributedIp, 0 failed. The live client run on the new head follows in a separate comment.

@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Live MCP client rerun at af0b7397, after the review fixes, on the same isolated setup: stack cwmcpfix on :8097, a TLS proxy on https://localhost:8496 as OAuth:Issuer, AllowedHosts=localhost and TRUSTED_PROXY_CIDR=172.16.0.0/12. The shared cluckwork-sim stack was not touched.

Consent. Two Owner tokens went through dynamic registration, sign-in, /connect with the password step-up, and the code exchange. One holds farm:read and the other both scopes. "Allow" with no password stayed on the consent screen. The read token lists the nine read tools, and the write token adds cluckwork_record_daily_entry.

A. Failure budget. The app was hit directly, with X-Forwarded-For from the trusted Docker gateway, so that two client addresses could be compared. Everything below ran inside one minute:

valid token x20 from 203.0.113.60, valid traffic only -> 20 x 200
forged burst from 203.0.113.60:     301 401
forged burst from 203.0.113.60:      29 429
valid token from 203.0.113.60 after the burst -> 429
valid token from 203.0.113.61 -> 200
valid token through the TLS proxy (127.0.0.1) -> 200

Valid calls spent nothing. The burst got one 401 more than the budget of 300, because 16 requests were in flight when the count crossed the limit. That is the check-then-record overshoot the PR body describes. After the burst the address is refused for the rest of the minute, valid token included, which is the accepted cost. Another address is unaffected.

D. Origin, every method, before authentication.

2025-11-25 POST no token, foreign Origin -> 403
2025-11-25 GET, foreign Origin -> 403
2025-11-25 POST token, null Origin -> 403
2025-11-25 POST token, issuer Origin https://localhost:8496 -> 200
2025-11-25 POST no token, no Origin -> 401
2025-11-25 GET, no Origin -> 405
2026-07-28 POST no token, foreign Origin -> 403
2026-07-28 GET, foreign Origin -> 403
2026-07-28 POST token, null Origin -> 403
2026-07-28 POST no token, no Origin -> 401
2026-07-28 GET, no Origin -> 405
2025-11-25 POST token, issuer Origin with a path https://localhost:8496/x -> 403

A raw 2026-07-28 tools/list carrying the issuer's Origin got past the Origin check and was answered 400 by the SDK. That curl lacks the per-request metadata 2026-07-28 requires. The SDK client (probe.cs) negotiated 2026-07-28 and 2025-11-25 and called cluckwork_read_stock on each without error.

J. tools/list shows openWorldHint: false on all nine read tools.

B. These calls recorded Sim Z Flock Catalog 001 on 2026-10-10 three times, each with a new key and each with replayed: false:

with lines     -> [{"grade":"Large","quantity":90},{"grade":"Medium","quantity":4}]
grades omitted -> [{"grade":"Large","quantity":90},{"grade":"Medium","quantity":4}]
grades []      -> []

C. The served idempotencyKey description reads: "A key you choose for this recording. A successful result is kept for at least 48 hours. Until then, resending the same call with the same key returns that result instead of recording again, and reusing the key with different values is refused. After 48 hours the key may be purged and a retry may record again. Before retrying an older call, check the current entry with cluckwork_list_daily_entries, or ask a person when this connection cannot read."

F. read_stock with limit: 1 returned Small, Medium, Large, Cracked and Dirty once each, ending complete: true, nextOffset: null. The fixture has one farm and no tied grades, so the tie case rests on the two tests.

The attached frame is the consent screen at 1280 px, light theme, 1:1, from this run.

Consent screen at 1280 px, light theme, with the password step-up, at af0b7397

mforce and others added 7 commits October 11, 2026 07:26
…uses writes

If Redis answers the read script but refuses increments, each increment lands in the in-process fallback and alarms, while every read returned the unchanged Redis count. CountAsync now adds the fallback's live count for the same key and window.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 301-request test assumed every request landed inside one real, wall-clock-aligned minute, so a run crossing a minute boundary saw 301 401s and no 429. These tests now run on an in-process counter over a FakeTimeProvider, and a separate test advances it to show a spent address recovers when the window rolls over.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DELETE, PUT and OPTIONS on /mcp select routing's own 405 endpoint, which carried no origin metadata and skipped the check. The middleware now matches the /mcp path as routing does, ignoring case with one optional trailing slash, independent of the selected endpoint.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refusals reach the shared failure budget only when they finish, so every lookup started concurrently was admitted. oauth-api endpoints now also cap bearer requests in flight per client address at 32, with no queue, refusing the next with 429. The cap is per process on purpose, as in-flight work is each instance's own, and the #271 record classifies it. The lease is released on every outcome.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e round-2 limits

The glossary said the key is forgotten after 48 hours; it is kept for at least 48 hours, after which it may be forgotten. The MCP endpoint record now names the in-flight cap and the path-keyed Origin check.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tead of hanging

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Reply to the round-2 Codex review at af0b7397. Each finding has its own commit, pushed as 3ca5b46d.

R6 (P2). Fixed in 7f31e807, with a test hardening in 3ca5b46d. oauth-api endpoints (only /mcp) now also cap bearer requests in flight per client address at 32, with no queue. The next bearer gets a JSON-RPC 429 with Retry-After: 1, logged under oauth-api-in-flight. It is a PartitionedRateLimiter concurrency limiter, per process on purpose, because in-flight work is each instance's own. The code comment and docs/decisions/271-single-serving-instance.md say so, so the next in-memory walk classifies it deliberately. A using lease covers the whole request, so it is released on every outcome, including exceptions and cancellation. The failure budget's ponytail: note now says its overshoot is at most this cap per replica.

BearersInFlight_FromOneAddress_AreCapped_AndAnotherAddressIsNot holds authentication open with an OpenIddict validation handler waiting on a gate:

  • 32 forged bearers from one address enter authentication, and the 33rd gets 429.
  • A bearer from 203.0.113.9 still enters authentication.
  • After the gate opens, all 33 held requests answer 401, and a further bearer from the first address gets 401, so the leases came back.

R7 (P2). Fixed in d35c3d7b. ResilientFixedWindowCounter.CountAsync adds the fallback's live count for the same key and window to the Redis answer. When Redis throws on the read, it returns the fallback's count and alarms, as before. CountAsync_WhenRedisReadsButRefusesIncrements_AddsTheFallbackCount uses a primary that answers reads (5) and throws NOPERM on increments. Two increments raise two alarms, and the read is 7.

R3 (P3). Fixed in 57077000. The Origin check now matches the /mcp path as routing does, ignoring case with one optional trailing slash, whatever endpoint routing selected. The issuer reaches it from Program.cs instead of endpoint metadata. ForeignOrigin_Gets403_BeforeAuthentication_OnEveryMethod adds DELETE and a foreign-Origin POST to /MCP/ for both revisions, and both get 403. Without Origin, DELETE gets 405, alongside the existing 401 and 405.

R8 (P3). Fixed in 0399a803. The failure-budget tests run on an in-process counter over a stationary FakeTimeProvider, through FrozenHost. The fake clock replaces only the counter, so token issuance keeps the real clock. This covers the 301-request test and both count-inspection tests. SpentFailureBudget_RecoversWhenTheWindowRollsOver spends 300 on that counter, gets 429, advances the clock one window, and gets 401.

Docs. be21669b changes the glossary to "kept for at least 48 hours, after which it may be forgotten". The OAuth runbook has no 48-hour sentence, and the in-app Help strings are untouched. The commit also records the in-flight cap and the path-keyed Origin check in the 806 decision record.

Mutations. Each mutation was applied to committed code, its tests run, and the file restored. All six went red:

Finding Mutation Red test
R6 the cap never refuses BearersInFlight_… (the 33rd request waited on the gate until the 30 s bound)
R6 the lease is never released (var instead of using var) BearersInFlight_… (the request after the gate opened got 429)
R6 one partition for every address BearersInFlight_… (the other address waited on the gate)
R7 the read ignores the fallback CountAsync_WhenRedisReadsButRefusesIncrements_AddsTheFallbackCount
R3 DELETE skips the check ForeignOrigin_Gets403_BeforeAuthentication_OnEveryMethod, both revisions
R3 trailing slash not matched the same test, both revisions

The targeted suite at 3ca5b46d passed: 440 tests across Mcp, RateLimit, OAuth, SharedState and DistributedIp, 0 failed. A live addendum for the burst, the real token and a foreign-Origin DELETE follows.

@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Live addendum at 3ca5b46d, after the round-2 fixes. Only the affected checks were rerun, on the same isolated setup: stack cwmcpfix on :8097, a TLS proxy on https://localhost:8496 as OAuth:Issuer, and an Owner farm:read token issued through real consent with the password step-up.

Origin on DELETE (R3).

2025-11-25 DELETE foreign Origin -> 403
2025-11-25 DELETE no Origin -> 405
2026-07-28 DELETE foreign Origin -> 403
2026-07-28 DELETE no Origin -> 405

Failure budget, in-flight cap and the real token (R6). The app was hit directly, with X-Forwarded-For from the trusted Docker gateway, all inside one minute:

valid token x20 from 203.0.113.70, valid traffic only -> 20 x 200
forged burst x330, 16 at a time, from 203.0.113.70:     303 401
forged burst x330, 16 at a time, from 203.0.113.70:      27 429
valid token from 203.0.113.70 after the burst -> 429
valid token from 203.0.113.71 -> 200
valid token through the TLS proxy (127.0.0.1) -> 200

The real token kept working from that address while valid traffic was all there was. The burst passed the budget of 300 by three requests that were already in flight, inside the cap of 32 in flight. After the burst the address was refused, valid token included, as the PR body describes, while another address was unaffected. The hold-open case for the in-flight cap is covered by BearersInFlight_FromOneAddress_AreCapped_AndAnotherAddressIsNot. A live burst cannot hold authentication open, so it is not shown here.

mforce and others added 2 commits October 11, 2026 08:20
…vershoot

The budget is read before the permit is taken, so any number of requests can pass the check before refusals land. The cap bounds the authentication work running at once, not the total overshoot.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the sum

An increment Redis applied before its reply timed out also landed in the fallback, so adding the two counts counted it twice. The larger of the two never double counts, still rises while writes fail, and undercounts by at most the smaller count.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es decide

The in-flight permit covered the whole MCP request, so 32 slow tool calls from one address refused the next authentication from it. The permit now goes back right after CredentialEpochMiddleware, before authorization, body reading and tool work, with no second authentication. The request's outer using still releases it on a refusal, an exception or a cancellation. The credential-gate order guard lists the new step.

Part of #789.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Reply to the round-3 Codex review at 3ca5b46d. Each finding has its own commit, pushed as 644bca77.

R9 (P2). Fixed in 644bca77. The permit is still taken before authentication. A new step, UseOAuthApiInFlightRelease, gives it back right after CredentialEpochMiddleware, once every credential gate has decided, and before authorization, body reading and tool dispatch. Nothing authenticates twice: the step only disposes the lease the earlier middleware stored on the request. The outer using still releases it on a refusal, an exception or a cancellation. A lease's Dispose releases its permit once, so the second disposal on the success path does nothing. CredentialEpochMiddlewareOrderTests pins the full sequence from authentication to idempotency, and now lists the new step.

AuthenticatedWorkInFlight_DoesNotHoldAuthenticationPermits holds a test endpoint's work open on 32 authenticated requests from one address. A fresh authentication from that address then gets 200. BearersInFlight_FromOneAddress_AreCapped_AndAnotherAddressIsNot still holds authentication itself open and still refuses the 33rd.

R10 (P3). Fixed in 73c4474f. CountAsync returns the larger of the Redis and fallback counts, not their sum. CountAsync_AfterIncrementsAppliedThenTimedOut_DoesNotDoubleCount uses a primary that applies each increment and then throws RedisTimeoutException. Three increments read as 3, not 6, with three alarms. The R7 test now has Redis answering 1 while two refused increments land in the fallback. It reads 2, which tells max (2) apart from Redis-only (1) and the sum (3).

R11 (P3). Fixed in ae667292. The ponytail: note now says the overshoot has no fixed bound, and that the cap bounds the authentication work running at once, not that total.

Mutations. Applied to committed code, tests run, file restored. All five went red:

Finding Mutation Red test
R9 the release step does nothing AuthenticatedWorkInFlight_DoesNotHoldAuthenticationPermits
R9 the release step removed from Program.cs the same test, and CredentialEpochMiddlewareOrderTests.Program_PinsTheCompleteCredentialGateSequence
R9 the permit released before authentication BearersInFlight_FromOneAddress_AreCapped_AndAnotherAddressIsNot
R10 the sum instead of the max CountAsync_AfterIncrementsAppliedThenTimedOut_DoesNotDoubleCount
R10 the Redis count only CountAsync_WhenRedisReadsButRefusesIncrements_AddsTheFallbackCount

The targeted suite at 644bca77 passed: 445 tests across Mcp, RateLimit, OAuth, SharedState, DistributedIp and the middleware order guard, 0 failed. A live rerun of the forged burst and the real token follows.

@mforce

mforce commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Live addendum at 644bca77, after the round-3 fixes, on the same isolated setup, with an Owner farm:read token issued through real consent with the password step-up. The app was hit directly, with X-Forwarded-For from the trusted Docker gateway, all inside one minute:

valid token x20 from 203.0.113.80, valid traffic only -> 20 x 200
valid token x200, 40 at a time, from 203.0.113.80:     200 200
forged burst x330, 16 at a time, from 203.0.113.80:     302 401
forged burst x330, 16 at a time, from 203.0.113.80:      28 429
valid token from 203.0.113.80 after the burst -> 429
valid token from 203.0.113.81 -> 200
valid token through the TLS proxy (127.0.0.1) -> 200

The real token kept working from that address while valid traffic was all there was, including 40 calls at a time, which is above the in-flight cap of 32. The forged burst ran two past the budget of 300 and was then refused. After it, the address was refused, valid token included, while another address and the proxy path were unaffected. A foreign-Origin DELETE still got 403 on both revisions, and DELETE without Origin got 405.

@mforce mforce mentioned this pull request Oct 11, 2026
16 tasks done
@mforce
mforce merged commit de61191 into main Oct 11, 2026
21 checks passed
@mforce
mforce deleted the fix/mcp-audit-hardening branch October 11, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant