Repository navigation
test(integration): run the OAuth and MCP test classes in parallel with the shared collection - #1286
Merged
Merged
Conversation
…h the shared collection Move OAuthFailClosedTests, the five MCP tool/authorization classes, OAuthServerTests and AccountLifecycleCommandTests off the serial IntegrationCollection onto their own IClassFixture factory, each with its own template-cloned database (#1002). No test body changes. summarize.py now matches partial classes, so it no longer drops OAuthFailClosedTests from the shared-collection census. Part of #1271
…a drain tests FailingUrl_IsFetchedOnlyWithinItsBudget and RemovedDocument_IsNotRevived_ByDrainingTheBudget counted against the host's wall-clock counter, so a loop that crossed a 5-minute bucket boundary let an eleventh fetch through. Their metadata hosts now take a frozen counter clock, as BudgetResolver and FrozenHost already do. Document expiry still reads the real clock. Part of #1271
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1271
Step 1 of #1271. Eight integration test classes leave the serial
IntegrationCollectionand each gets its ownIClassFixture<CluckworkWebApplicationFactory>, the pattern eight classes already use. With #1002 each factory gets a template-cloned database. The serial collection is one thread and decides when the integration leg ends, so these classes now run on the idle cores instead.Moved:
OAuthFailClosedTests(5 partial files),Mcp.RecordDailyEntryToolTests,Mcp.McpAuthorizationTests,Mcp.McpToolErrorTests,Mcp.McpReadToolsTests,Mcp.McpCommerceToolsTests,OAuthServerTests,AccountLifecycleCommandTests.No assertion or test body changes. Each moved class changes in two lines: the
[Collection]attribute goes and the fixture interface comes in.Two smaller changes:
tools/test-timing/summarize.pynow matchespartialclasses. The old pattern droppedOAuthFailClosedTestsfrom the shared-collection census.Audit: why each class can leave the serial collection
OAuthFailClosedTestsHost(),OAuthFailClosedTests.cs:766), soIFixedWindowCounter, OpenIddict state and theProbefilter belong to that host. The factory configures no Redis, only a key namespace per factory.MetadataDocumentServerstarts per test on port 0, andBudgetResolvergives each global-budget test a new counter. The one read outside its own tenant is the default account'sAllowConnectedAppsinConnectedApps_AreOnByDefault…. No other class writes that value, and a fresh clone holds the migration default that the test asserts. Other counts filter by the test's own subject, client id or URL.Mcp.RecordDailyEntryToolTestsMcpConnection.Host). Each test seeds its own farms. Every count is tenant-scoped or filters by the test's own account or flock id, including theIdempotencyRecordscount. In its own database, the global purge sweeps of other classes cannot reach its rows.Mcp.McpAuthorizationTestsAllowedHosts=*at:140). Own farms.Mcp.McpToolErrorTests:156). Own farms.Mcp.McpReadToolsTestsAssert.Emptychecks (:39–41) are MCP reads by the test's own tenant. A foreign farm it seeded must not appear. The reads are tenant-filtered, so an empty result means the same in any database.Mcp.McpCommerceToolsTestsOAuthServerTestsclient-{Guid}(:248). The token read at:55filters by the test's own user id.OAuthClientRegistrationTestsstays in the shared collection and callsnew OAuthServerTests(factory)with its own factory, so the two classes share helper code, not state.AccountLifecycleCommandTestsConnectionStrings__Default = factory.ConnectionString(:25), the class's own cloned database. They are run-then-exit verbs with no port and no hosted services. Slugs come from the seeded account id.missing-farmis a deliberate unknown code. All counts filter by the test's own account id.Shared state checked for all eight:
src/has no mutable process-wide statics.TestHarness.cs:29. It is keyed by a random token, so classes cannot collide in it.EnvironmentMutatingCollectionwrites environment variables and disables parallelization.QueryShapeWarningInitializer.cs:9writes one at module initialization, before any test runs.Lazystatics.DurableJobWorkerleader lock and global sweeps see only that class's rows.#863 §D2 lists the seven classes that depend on shared-database rows. None of these eight is among them, and a grep agrees: the only
SeedDefaultsreference is the read described above.Frozen counter clock for the two per-URL budget tests
FailingUrl_IsFetchedOnlyWithinItsBudgetandRemovedDocument_IsNotRevived_ByDrainingTheBudgetcounted fetches on the host's real counter. That counter's 5-minute buckets align to the wall clock. If a loop crosses a bucket boundary, an eleventh fetch passes and both assertions fail. Parallel load makes the loops slower, so this PR fixes it.MetadataHostAsynctakes an optionalcounterClock. Those two tests pass aFakeTimeProvider, and the host'sIFixedWindowCounterbecomes anInProcessFixedWindowCounteron that clock. #1258 used the same mechanism for the global-budget tests. Document expiry still reads the real clock. No production window changed, and no retry was added.Mutation proof. The mutations were temporary and none is committed.
Local results
All runs were on the shared 12-core host with no other test stacks running.
dotnet build: 0 warnings, 0 errors.--filter "FullyQualifiedName~OAuthFailClosedTests|FullyQualifiedName~Mcp.|FullyQualifiedName~OAuthServerTests|FullyQualifiedName~AccountLifecycleCommandTests", 3 runs: 334/334 each time, in 2m46s, 2m50s and 3m42s. In the trx, all eight classes start together at 11 s.OAuthClientRegistrationTests, after the fix: 378/378, twice.pg_stat_activity: 58 to 65 againstmax_connections=208. That is headroom on a filtered run, not a worst-case bound for the full suite.CI A/B
Method from #863 §U7. Only the
Tests (integration)job was rerun, withgh run rerun --job.92bdee8c, attempts 1 to 11.391036d0, this PR's base. Main had only one push run since ci: record per-test timing and runner CPU for the integration leg #1272, and rerunning main's integration job would also rerunpublishand push:sha-391036d0again. Both sides therefore ran aspull_request.McpToolErrorTests(see below), so attempt 11 replaced it.All runners had 4 vCPUs. Seven CPU models appeared, in two speed tiers. Times are in seconds, median (range).
On the one CPU both sides drew often, AMD EPYC 7763, the leg went from 1013 s (993–1025, n=7) to 731 s (730–766, n=5), 282 s or 28% less, and the two ranges do not overlap. Fast runners appeared too rarely on either side for a median to mean much. Per CPU: EPYC 9V45 went from 664 s to 514 s, and EPYC 9V74 from 840 s to 586 s. The one main push run since #1272 (38292467802, EPYC 9V45) took 676 s, with the serial collection alone for 320 s.
The shared collection no longer runs alone at the end: its solo time is 0–5 s on every PR sample. The leg now ends with the rest of the parallel work. Fast runners take 8.6–9.8 min, which is not yet the 6–7 min that #1271 aims for. Step 2 (reusing one host per class) cuts total CPU work and is the next lever.
Failure seen in the A/B (already existing, not fixed here)
Attempt 9 failed
McpToolErrorTests.BadArgument_IsNamedWithItsShape_AndLogsNoError(job 115013721759). The tool refusal was correct (IsError = True, logged at Information). Then the/mcpPOST of that same call ended in aTaskCanceledExceptioninStreamableHttpPostTransport.HandlePostAsync, and Serilog's request log wrote it as a 500 at Error. This is the #1246 failure. #1249 drops late events from earlier requests by trace id. This event has the call's own trace id, so the filter keeps it. The likely cause is the MCP client closing the response stream after it reads the result, before the server finishes the POST. Extra CPU load from running in parallel makes that window wider. One failure in 11 PR samples, none in 10 baseline samples.