Repository navigation
Description
The main Performance run reports 749 B/message for ConcurrentRoundTrip against bracketed baselines of 747 and 748 B/message. The strict allocation comparison fails; the allocating owner and whether this depends on the candidate remain unknown.
Reproduction
Download artifact performance-43367252770-1 (14011632127) from https://github.com/thomhurst/Relayer/actions/runs/43367252770 and replay the reports with .github/scripts/benchmark_gate.py compare and its input manifest. Candidate: 4a6ff1d. Failed step: Compare bracketed throughput, allocations and startup.
Expected and actual
ConcurrentRoundTrip should not exceed both baselines by more than 0.5 B/message. Candidate throughput confidence intervals overlap both baselines; DirectSend stays 0 B/message and SynchronousRoundTrip stays 536. The comparison lists only the added frozen-net10-protocol-control workflow, but unchanged call-path source does not establish identical build inputs or execution. No infrastructure failure, accepted trade-off, rerun or threshold relaxation is established.
Environment
Hosted Linux x64; exact runtime, executable inputs and original reports are in the retained artifact. This is a distinct execution from #537 and #488; those investigations do not attribute this signal.
Current investigation status
Baseline-only same-image run 43380948169 succeeded. All 711 frozen inventory entries and 849 retained files / 290,588,136 bytes were independently length/SHA-256 verified. All four launches use the same original baseline executable/job and runtime 10.0.15. GC B/message is 726.835205 / 724.988281 / 730.366943 / 728.705322, with 32,768 native operations each. The 5.378662 B/message spread occurs without changing executable selection in this new instrumented run; it does not attribute the historical signal.
All 296 tail stack groups and complete native histories were inspected; zero lost events, missing stacks or unnamed frames. Sampled owners include native channel pending reads, fixture pooled handlers/Actions, DI storage, serializer state, fixture objects and one native ArrayPool finalizer GCMemoryInfoData tick. All previous adverse results and verified inventories remain retained. Samples are not exact counts; tail includes cleanup/stat setup; instrumentation, process order, adaptive timing, host drift and historical loaded-byte identity remain limits. No production defect is demonstrated.
Native subissue #559 is closed through #561. Exact native-tail callback tooling in merged PR #558 is available on main; its completed executions are summarized below and do not themselves deliver historical attribution for #555. No duplicate tooling was added.
Candidate-only same-image run 43381933998 also succeeded. All 711 frozen inventory entries and 849 artifact files / 296,247,122 bytes were independently verified. Four GC B/message results are 744.185547 / 745.201172 / 744.613525 / 751.221924, each with 32,768 native operations and one Gen0 collection. The 7.036377 B/message spread occurs with original candidate executable selection fixed. All 257 tail stack groups and all native event histories were analyzed with no lost events or missing stacks. Native ArrayPool finalizer GCMemoryInfoData samples in launches 2 and 4 are retained. Neither this variation nor the collection difference against the independent baseline control attributes the historical signal. Do not subtract independent arm totals or infer exact object counts from weighted samples. Full evidence and limits: candidate-control analysis.
The earlier blocker is resolved: #558 merged as 9bc20f48410d57599fadaeea36dc59891df01062 and required CI 43384281860 passed. Trusted main now includes its exact native-tail callback tooling.
Exact-tail run 43388948971 succeeded. All 711 frozen entries and 942 retained files / 230,533,890 bytes were verified. Native ABBA GC B/message: 739.182617 / 741.649414 / 735.216797 / 735.553955; 32,768 operations each, zero callback failures. All 45 type families inspected, including adverse candidate 2 and favorable candidate 3. Fixture handler pooled boxes/Actions and native channel-read counts vary; constant DI/serializer/message families remain retained. Exact callback counts describe this new tail envelope, not historical ownership; tail cleanup/stat setup, instrumentation, process order, drift and loaded-byte identity remain limits. Full evidence is in the latest analysis comment. No production defect is demonstrated.
Method-boundary run 43391560343 succeeded. All 711 frozen entries and 927 retained files / 230,508,145 bytes were independently length/SHA-256 verified. Native ABBA GC B/message: 728.419312 / 722.525696 / 723.333008 / 724.652466 with 65,536 / 131,072 / 262,144 / 65,536 native operations. Every launch has one exact MeasureWithGc entry/normal leave and zero profiler failures; all 13 type families were inspected. Both candidates are below their baseline bracket; fixture handler pooled boxes/Actions and native channel reads vary. This removes the broader tail envelope but does not attribute historical ownership. ReadInitial/ReadFinal, concurrent background work, instrumentation/JIT/scheduling, adaptive counts, process order, drift and historical loaded-byte identity remain limits. Full analysis is in the latest issue comment.
Exact-method baseline-only run 43394508094 succeeded. Independently retained/read-back hash verified all 927 files / 230,491,172 bytes and all 711 frozen entries. Actual logs/protocol select Performance 43367252770, runtime 10.0.15 and four original baseline images. GC B/message: 734.470276 / 735.497009 / 739.560303 / 738.447327 at 131,072 operations each. Each report has one method entry/normal leave and zero failures. All 14 type families were inspected, including the single background callback. Identical-image variation here does not attribute the historical signal or prove it was noise; all prior unfavorable evidence remains retained.
Candidate-only exact-method run 43396668579 succeeded. All 711 frozen entries and 927 artifact files / 230,491,169 bytes were independently SHA-256/length verified; all fourteen prior/current retention manifests reverified. Actual logs/protocol confirm Performance 43367252770, runtime 10.0.15 and four candidate selections. Native GC B/message: 733.896973 / 728.538879 / 736.448242 / 731.401489, operations 131072 / 131072 / 65536 / 131072. Every method boundary and complete fourteen-family callback report validates with zero failures, including the two background callbacks. Complete counts, evidence and limits are in the latest investigation comment.
The console-parent allocation controls and bounded native-parent JIT comparison are complete. A distinct child-only exact-owner capture with the original BenchmarkDotNet pipe parent is now complete: 43513326001; see the bounded follow-up below. The original retained artifact has aggregate measurements and executable inputs, but no original allocation-owner events/stacks. New controls identify owners only in their own instrumented executions; historical attribution remains unresolved and no candidate-specific production defect has been demonstrated. Repeating an unchanged control cannot reconstruct events never recorded. No repeated allocation control, speculative production fix, Performance rerun, relaxed threshold or accepted trade-off is proposed. Original failure and adverse evidence remain preserved. The historical criterion remains explicitly unmet; the completed bounded investigation and durable-retention disposition are recorded below.
Completed bounded JIT comparison
Merged #694 (8a4ccbbd71fb6ecc35ab1d5b1fa6d25a1e1850eb) adds child-only native JIT capture to the existing frozen BDN launcher. Run 43451796148 succeeded. Actual job environment, protocol, native receipts and logs independently confirm Performance 43367252770, concurrent original runner 10, native-parent ABBA and runtime 10.0.15. All 711 frozen entries and all 844 artifact files / 262,136,662 bytes were SHA-256/length verified; all fifteen old/new retention inventories reverified. Hosted launcher tests: 25 passing.
Every raw handler MoveNext listing has three complete observed tiers: Tier0 (677 code bytes), instrumented Tier0 (832), and Tier1 (1358 / 1358 / 1362 / 1360 for baseline/candidate/candidate/baseline). Tier0 and instrumented Tier0 text match across all arms. First candidate Tier1 differs from the first baseline only in reported PGO count. The second candidate has reordered resume/completion blocks and direct/indirect bulk-write-barrier call differences; baseline 4 also differs from baseline 1 in completion block layout/registers and bulk-write-barrier calls. All six pairwise raw diffs, full tier hashes and all original listings are retained. Neither favorable nor adverse differences were filtered away.
No malformed/interleaved or truncated listing was observed in these twelve bodies. This is filtered observed-method coverage, not proof that no method was missing, inlined, precompiled or compiled at an unobserved tier. New listings cannot reconstruct historical code, identify the historically allocating owner, prove which tier executed during measurement, or justify a production change alone. All diagnostic timing and allocation values from this JIT run are excluded. The original failure and all earlier confounders remain. No further experiment was dispatched; the original missing owner events cannot be recreated by executing retained images again. The remaining historical-attribution criterion is explicitly unmet, and no production defect has been demonstrated to correct.
Remaining acceptance criteria
- Retain/hash all native-tail artifacts and frozen inventories; validate every native outcome/runtime/event history and tail denominator. Runtime manifests alone do not retain runtime bytes or establish historical loaded identity.
- Analyze all four native adaptive outcomes and every tail owner/stack group and missing coverage, preserving unfavorable/failing outcomes. Compare each candidate only to its own bracket; do not subtract independent totals or call sampled weights exact counts.
- Attribute the signal through controlled retained-image comparisons and complete owner evidence. Preserve profiler, sequential-arm/process-order, drift, IL/metadata substitution and fixed-count protocol confounders. Completed controls are not historical object attribution.
- Correct any demonstrated defect with focused coverage while preserving unfavorable results and inference limits.
Please identify the allocating owner and correct any demonstrated defect.
Bounded native-parent owner follow-up
The earlier exact-method allocation reports explicitly use Native NoAcknowledgementConsoleHost; original anonymous-pipe parent absent. The completed native-parent run 43451796148 captures JIT listings, not exact allocating owners. Merged #607/#610 now support the original BDN anonymous-pipe parent with profiling enabled only in its benchmark child. This removes a distinct parent/profiler perturbation from the controlled owner question; it does not recover the original historical allocation stream.
Run 43513326001 was dispatched once on trusted main cb890c0e1397dba4e77eb6df4bf12ba4eaa97250, with explicit performance_run=43367252770, parent=native, same_image=comparison, runtime_version=10.0.15. No subsequent exact-owner run among 47 inspected logs already selected these Performance inputs. Reuse the merged profiler, native BDN parent, original generated executable/Default job and method boundary; no replacement profiler, modified workload or Performance-gate rerun.
Completed: all 711 frozen entries and 1,009 artifact files / 264,563,344 bytes verified; all four native receipts/GC records/completions and exact method reports validate. Every one of the thirteen type families was analyzed. ABBA native B/message: 732.715393 / 732.941162 / 733.710693 / 734.352295. Both candidates lie inside the baseline bracket in this execution. Full raw counts, sizes, normalized owners, adverse outcomes and inference limits are in the completion comment linked below; no cross-control subtraction or historical-causation claim.
The actionable question is complete present-day mechanism attribution on the retained images under the original parent; exact historical ownership remains unknown. The original acceptance criterion is retained above and is not checked off by dispatching this run. The transparent scope disposition used in linked #537 is available once this bounded evidence is analyzed and all original/adverse evidence is preserved independently of local cleanup. This is not proof the historical failure was noise or a passing gate. Correct only a demonstrated defect; none is established by existing results.
Completed bounded investigation and scope disposition
Completion evidence and full Resolve audit closes the previously recorded present-day mechanism question against origin/main 238c8f2e7025b2a254c0c4d839855de5d3cd279a. Native child #559 is closed; #561/#607/#610/#694 provide the reusable implementation. All captured callback families are accounted for: stable DI/serializer/fixture objects contribute 536 object-size B/message; native channel waits and fixture pooled handler boxes/Actions contribute the variable remainder. No demonstrated candidate-specific production defect exists to correct.
Historical allocation ownership remains unknown and the original historical criterion above is not checked off. The aggregate original artifact lacks owner events, which no new execution can recover. This resolution follows the transparent investigation-scope decision described above and in #537; it does not call the original strict failure noise, passing, or causally explained. No speculative production patch, relaxed threshold or performance trade-off was made.
All original/adverse artifacts and analyses are preserved independently of local cleanup and Actions expiry in unpublished draft release performance-evidence-555, ID 444777456: content-addressed archive of 15,398 paths / 4,341,552,590 original bytes, with read-back-verified path/length/SHA-256 manifest. Archive digest b91736b720fdf10c3a48d9166de95bc3eaa27aa78f82e8752622640bbcaf71d3, 64,119,958 bytes, matches GitHub's asset digest. Maintainer authentication is required; this draft evidence storage remains until explicitly deleted. Local evidence is disposable. No docs impact or product source change.
Activity
thomhurst commentedon Nov 13, 2026
The quoted earlier blocker is resolved: #545 merged and CI 43353026897 succeeded. Inspected failed main Performance 43367252770: only ConcurrentRoundTrip allocation comparison fails (747/748 baseline, 749 candidate B/message); no infrastructure failure or rerun is justified. Searched open/closed issues and open PRs: none delivers or mentions this execution. #537/#488 investigate distinct retained executions, and #348 changes test-process GC settings only, not benchmark execution. No merged fix, superseding PR, finished parent, stale acceptance criterion, human decision, credentials or upstream gate solves this failure. Filed/claimed this distinct bug instead of deferring main again; released main-ci (confirmed FREE).
Completed independent original retention at
/Users/tom/git/Relayer-evidence/issue-555/original/: all 711 frozen inventory entries match lengths/SHA-256; all 772 files / 228,297,110 bytes read-back verified againstoriginal-retention-sha256.jsonoutside removable worktrees. The original strict unfavorable report remains unchanged. Baseline 7554d95; candidate 4a6ff1d; native BDN 0.15.8 / SDK 11.0.100. Reported input difference is the added diagnostic workflow; source identity alone cannot establish unchanged generated binaries or execution. Local retention lasts while this issue is open.Started one existing trusted-main retained-image diagnostic, not a Performance gate rerun:
gh workflow run frozen-net10-protocol-control.yml --ref main -f performance_run=43367252770, run https://github.com/thomhurst/Relayer/actions/runs/43370064606. Verified workflow_dispatch, main, exact current main SHA 4a6ff1d and expected workflow path. It is in progress; do not start a competing execution. This reuses native BDN adaptive execution and the existing original-waiter fixed-count control without changing frozen images or production code. A same-host bundled protocol comparison is the next available route before proposing a fix for an unattributed signal; this macOS ARM64 host cannot execute retained Linux x64 images.Next unit: retain/hash every hosted artifact and frozen inventory, inspect all ABBA outcomes, native operation denominators and execution identity, and compare candidates only within each arm's baseline bracket. Process order, drift and protocol differences remain confounders; no cross-arm subtraction, historical owner attribution, exact type counts, production defect, threshold relaxation or accepted trade-off is claimed. Choose owner-specific profiling from those outcomes if needed. No demonstrated code defect can yet be safely fixed. No on-hold/needs-human label applies to this temporary run wait. No owned services started; releasing issue-555 and stopping this unit without waiting for hosted completion.
thomhurst commentedon Nov 13, 2026
Re-checked and reclaimed #555 under the driver-owned issue lock. The issue remains open, unassigned and has no blocking dependency, open PR, or local issue-555 branch/worktree to publish. Its earlier work is retained investigation evidence rather than unpublished source. Independently verified all 772 original retained files against their recorded lengths and SHA-256 hashes at
/Users/tom/git/Relayer-evidence/issue-555/original-retention-sha256.json.The existing trusted-main retained-image diagnostic https://github.com/thomhurst/Relayer/actions/runs/43370064606 is still running: original adaptive BenchmarkDotNet ABBA completed successfully; the fixed-count original-waiter ABBA is in progress and artifact retention has not executed yet. This is an execution wait, not a human/upstream blocker. No competing run, gate rerun, production change, allocation-owner inference, or threshold relaxation is justified before its complete retained outcomes exist.
Next unit after that run completes: retain and hash its artifacts and frozen inputs, analyze every ABBA result and operation denominator within its own bracket, then select owner-specific profiling or fix any demonstrated defect. Original unfavorable results remain preserved. No services started. Releasing issue-555; no on-hold or needs-human label added.
thomhurst commentedon Nov 13, 2026
Triage against fresh
origin/main(4a6ff1d9892a86aabbaad8d27152cc261aa4ea28): diagnostic 43370064606 completed successfully. Updated the issue body with its artifact identity and remaining work; the previous execution wait is resolved. Next implementation unit should retain/hash that artifact, verify frozen inventories, analyze every outcome/denominator within its own bracket, then select owner-specific profiling or fix a demonstrated defect. No artifact analysis, allocation-owner inference, production fix, rerun or threshold change is claimed by this triage.No native blocker/sub-issue, open delivery PR, or local issue-555 branch/worktree backs an implementation claim. Clearing stale
in-progressas the triage lock is released; no hold or human decision is needed. Original issue scope and P2 priority remain valid. No services started.thomhurst commentedon Nov 13, 2026
The earlier wait is resolved: protocol control 43370064606 completed successfully. Re-checked #555: no open delivery PR, local issue-555 implementation branch/worktree, native blocker or sub-issue existed. Claimed the issue under the driver-owned Redis lock; there is no stranded source to publish, shipped fix to close against, finished parent, superseding PR, or stale remaining scope to remove. This unit created an isolated investigation worktree and made no production changes.
Independently retained artifact
frozen-net10-protocol-control-43370064606-1(14012196912) at/Users/tom/git/Relayer-evidence/issue-555/protocol/. All 711 frozen inventory entries verified (237 per baseline/candidate/baseline-after). All 826 artifact files / 228,748,784 bytes were hashed and independently read-back verified againstprotocol-retention-sha256.json. Reverified all 772 original files againstoriginal-retention-sha256.json; the original unfavorable reports and executable inputs remain unchanged. Local retention lasts while this issue remains open.Reparsed native GC lines from all four adaptive logs and checked against their outcomes. ABBA results in B/message:
Adaptive allocated bytes are 48,493,920 / 48,548,848 / 48,496,328 / 48,655,296, each divided by 65,536 native total operations (already includes eight operations/invocation). Fixed-count bytes are 196,527,264 / 195,512,360 / 195,149,784 / 196,166,720, each divided by 262,144 messages (32,768 batches times eight). Every launch completed. Adaptive candidates lie within their own baseline bracket; fixed-count candidates lie below both of theirs. Neither arm reproduces the original unfavorable direction. No cross-arm subtraction, historical allocation attribution, exact type counts, proof of no defect, or budget/threshold relaxation follows. Separate processes, adaptive-before-fixed ordering, host drift, and bundled protocol differences remain confounders.
Next available route is owner evidence, using existing native BenchmarkDotNet replay, pinned dotnet-trace gc-verbose capture, and whole-process FrozenAllocationReader rather than another inclusive control. Checked recent owner captures: 43370069876 and 43367971899 target 42760437053, not this issue's run. Started exactly one trusted-main owner capture for 43367252770: https://github.com/thomhurst/Relayer/actions/runs/43370818067 . Verified workflow_dispatch, main, SHA 4a6ff1d, and workflow path frozen-native-owner-control.yml. This is a new instrumented owner-family experiment, not a gate rerun; sampled whole-process owners include startup/pilot/warmup/GC-stat work and cannot identify exact objects or reconstruct the historical 1–2 B delta. Do not start a competing run.
Next unit: retain/hash its complete artifact and inventories, inspect every native outcome and whole-process owner stack (including unfavorable or failed outcomes), then choose a targeted retained-image intervention or correct a demonstrated defect. The completed protocol control alone does not justify a code fix. Existing native/Relayer tooling already supplies this route; no parallel implementation is needed. No user-facing behavior/docs impact and no per-message production cost. No human, upstream, credentials or locked blocker remains; only the newly started hosted execution is pending. No on-hold/needs-human label or unrelated unit. No owned services started.
thomhurst commentedon Nov 13, 2026
The earlier blocker is resolved: owner capture 43370818067 completed successfully. Re-checked and claimed #555 under the driver-owned issue lock: still open/unassigned, no native dependencies/sub-issues or open delivery PR. No existing issue-555 source branch/worktree was stranded. Fresh origin/main remains 4a6ff1d; no shipped fix, superseding PR, finished parent or stale acceptance criterion closes the remaining attribution work. No human, upstream, credential or other-worker blocker was found.
Retained owner artifact
frozen-native-owner-control-43370818067-1(14012467434; 95,197,477 compressed bytes) at/Users/tom/git/Relayer-evidence/issue-555/owner/. Verified all 711 frozen inventory entries and independently read-back verified 845 files / 326,482,102 bytes againstowner-retention-sha256.json. All original 772 and protocol 826 files were reverified against their existing inventories; original unfavorable reports remain unchanged. Full owner-family aggregation and representative stacks are retained inowner-analysis.json. Retention lasts while this issue remains open.Reparsed every raw native GC line against outcomes.json. In ABBA order:
Native operations already include eight messages per invocation. Both candidates lie inside their own baseline bracket; no historical reproduction or absence-of-defect proof follows. All four processes use reported .NET 10.0.15 (also reported by the original execution), but runtime/module events and PDB signatures do not prove loaded binary bytes or historical execution identity. Process order, drift, profiler attachment and original-parent absence remain confounders.
Inspected every owner row, aggregated into 96/89/104/102 owner/stack families. Across all four captures the material owners include DI scope/dictionary/entry arrays; JSON argument state and message creation; native bounded-channel blocked reads for byte/InMemoryMessage/int; and the ConcurrentHandler pooled state-machine box plus its MoveNext Action. Byte-channel stacks reach the handler's pending release read and the workload's entered signal; int-channel stacks reach the workload completion read. Whole-process startup also appears and cannot be assigned to the measured region. No Relayer core pooled-box type was sampled, which is not evidence of zero allocations. Sampled ticks for handler boxes are 1816/1775/2528/1563, byte-channel reads 3075/2963/4126/2379 and Actions 854/790/1134/682; these are sample ticks, not exact objects or a subtractable delta. Zero events lost in each capture; B1 has one stackless sample, the others zero. Whole-process capture includes startup, pilot, warmup, overhead and GC-stat work with no measurement markers.
The next actionable route is to separate the observed handler/channel/driver families with existing exact synchronous regional probes. Started exactly one trusted-main
frozen-net10-owner-control.ymldispatch for 43367252770: 43371395168. Verified workflow_dispatch, main, expected path and SHA 4a6ff1d. Recent existing executions predate this target and do not replace it. This reuses FrozenConcurrentProbe's original BDN waiter, FrozenFixtureProbe's source-correlated handler/native-read counters, and FrozenDriverProbe's worker and calling-thread counters; no parallel implementation or production modification is needed. Each region has its own ABBA; never subtract different controls. Wrappers, retained identities, fixed counts and instrumentation change scheduling, so exact regional byte counts still do not reconstruct historical objects.Next unit: retain/hash every artifact and input inventory from 43371395168, inspect all four outcomes for each arm (including failures/unfavorable results), then select a causal retained-image intervention or fix a demonstrated defect. No speculative code fix, threshold relaxation, gate rerun, accepted performance trade-off or docs-impacting behavior change is justified yet. Single-unit mode stops after starting this hosted diagnostic; no competing execution, on-hold/needs-human label or unrelated unit. No services started. Releasing issue-555 and its clean isolated worktree.
36 remaining items
thomhurst commentedon Nov 13, 2026
Earlier wait is resolved: 43396668579 completed successfully. Rechecked driver-owned issue-555 (HELD-BY-ME), cleared stale in-progress and reclaimed it. Fresh origin/main is c9cfc7f; all twelve earlier issue-555 branches are ancestors, no earlier issue-555 worktree or unpublished work exists, and no open delivering PR exists.
Retained complete artifact
frozen-exact-adaptive-43396668579-1(14023366558; 82,288,800 compressed bytes) at/Users/tom/git/Relayer-evidence/issue-555/exact-method-candidate/frozen-exact-adaptive-43396668579-1/. Independently verified all 711 frozen inventory entries and read-back SHA-256/length verified all 927 files / 230,491,169 bytes. Reproducer and every type/outcome:analyze-exact-method-candidate.py,exact-method-candidate-analysis.json/log,exact-method-candidate-retention-sha256.json. Reverified all fourteen retention manifests (including original and all adverse experiments); counts are recorded inmethod-candidate-prior-inventories.json. This is local evidence retained while the issue is open, not a permanent upload.Actual execution log and protocol independently confirm PERFORMANCE_RUN=43367252770, CONTROL_RUNTIME=10.0.15, SAME_IMAGE=candidate and four original candidate selections.
Every raw native GC record reconciles with its outcome. All four exact reports have one MeasureWithGc entry and normal leave, zero callback/metadata failures and reconciled complete type totals. Every logged runtime is 10.0.15. All fourteen type families were inspected; complete callback object counts follow (launches 1 / 2 / 3 / 4):
Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScopeRelayer.Benchmarks.ConcurrentHandlerRelayer.Benchmarks.ConcurrentOrderSystem.ActionSystem.Collections.Generic.Dictionary2+Entry[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey,System.Object][rank=1]`System.Collections.Generic.Dictionary2[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceCacheKey,System.Object]`System.Int32System.Int32[rank=1]System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1+StateMachineBox1[System.Threading.Tasks.VoidTaskResult,Relayer.Benchmarks.ConcurrentHandler+<HandleAsync>d__2]System.Text.Json.ArgumentStateSystem.Threading.Channels.BlockedReadAsyncOperation1[Relayer.InMemory.InMemoryMessage]`System.Threading.Channels.BlockedReadAsyncOperation1[System.Byte]`System.Threading.Channels.BlockedReadAsyncOperation1[System.Int32]`System.Threading.QueueUserWorkItemCallbackDefaultContext1[System.Object]`Constant families occur once per operation. Fixture handler pooled state-machine boxes and matching Actions vary at 0.349945 / 0.336678 / 0.377625 / 0.334610 objects/message. Native byte-channel blocked reads vary at 1.332054 / 1.304169 / 1.301208 / 1.338249. The message/int channel read variation and two individual background callbacks (launches 2 and 4, 32 bytes each) are preserved. No core pooled-box family appears here; this does not establish historical absence.
The unchanged candidate spans 7.909363 B/message in this separate instrumented control. This describes variability here; it does not identify the original 749 versus 747/748 allocating owner, prove the original failure was noise, or authorize subtraction of independent controls. Preserve all prior adverse results and the callback/metadata/locking, ELT/JIT, ReadInitial/ReadFinal, concurrent activity, adaptive counts, sequential process order, host drift and historical loaded-byte identity limits. Object sizes exclude allocation-context/alignment overhead. No throughput claim, gate relaxation or performance trade-off is established.
Finite investigation conclusion: the original retained artifact contains aggregate GC/BenchmarkDotNet reports and executable builds, but no historical allocation events/stacks/object-owner trace. Those historical events cannot be recovered by executing the images again. The completed controlled executions identify their own variable fixture/runtime owners, but demonstrate no candidate-specific production defect to correct. Another identical control cannot fill that historical evidence gap; no further run was dispatched and no speculative code fix was made. The historical attribution criterion remains unmet; this comment does not claim implementation or resolution.
Resolve audit: the earlier run blocker no longer holds; no current blocker issue/PR can be worked to recover events never captured. Closed native child/prerequisite #559 and merged #561/#558/#589 provide diagnostic tooling, not historical attribution. There is no completed-parent acceptance, open superseding delivery PR, already-shipped attribution/fix, recoverable unpublished branch, stale partial implementation that completes the remaining criterion, reversible implementation choice that creates the missing history, or human/upstream/credentials/foreign-lock gate. Retain the original strict failure and remaining criteria. Leaving this issue open also preserves its full local evidence under repository cleanup policy; not closing as fixed/unconfirmed on ephemeral evidence. This unit reports failure on missing historical evidence rather than another unchanged wait/deferral.
No source/user-facing change, docs impact, production per-message cost or unrun-test claim. Analysis used the existing native BDN/profiler reader; no duplicate profiler. No owned services started. Removing the in-progress claim and releasing issue-555 and its clean isolated analysis worktree.
thomhurst commentedon Nov 13, 2026
Rechecked and claimed #555 under driver-owned issue-555 (HELD-BY-ME) against fresh origin/main
2a2eb41d9d6f3a3c570fbf36d9dde736c34f1cad. The sole native child and prerequisite #559 is closed, and #561/#558/#589's reusable controls are on main. The parent's first two acceptance criteria are completed; historical owner attribution remains unmet, and no candidate-specific production defect has been demonstrated to correct. All original 772 files / 228,297,110 bytes were independently read-back length/SHA-256 reverified againstoriginal-retention-sha256.json. That inventory contains aggregate reports/executables, no original allocation-owner trace. New execution cannot reconstruct those missing historical events.Resolve audit: no open dependency to work, no open delivering/superseding PR, no unpublished #555 branch/worktree or shipped historical attribution/fix, and closing child #559 does not complete this parent's remaining criterion. The unchanged historical requirement cannot truthfully be checked off or converted into proof of noise. No human, upstream-release, credential or foreign-lock blocker exists. Corrected the stale body reference to open #558; it is merged. No repeated allocation control, speculative production patch, threshold relaxation or accepted performance trade-off.
There is a new route since the earlier finite investigation: merged #694 (
8a4ccbbd71fb6ecc35ab1d5b1fa6d25a1e1850eb) adds child-only native JIT capture to the existing launcher. This can answer a narrower unresolved execution question: do the retained images produce different observed bodies/tiers for the fixture handler state machine, whose pooled-box/Action counts varied in the complete exact callback controls? Reused that native .NET diagnostic and existing launcher rather than creating another profiler. Recent dispatch logs were checked: none of the post-investigation frozen-bdn runs targeted Performance 43367252770; recent JIT runs target #613's different inputs.Started exactly one trusted-main native-parent ABBA capture: 43451796148. Submitted inputs:
performance_run=43367252770,benchmark_case=concurrent,parent=native,same_image=abba,runtime_version=10.0.15,jit_method=*ConcurrentHandler*:MoveNext. API receipt verifies workflow_dispatch/main/expected workflow path and head2a2eb41d9d6f3a3c570fbf36d9dde736c34f1cad. Inputs and receipt are retained asfinal-audit-jit-dispatch.jsonandfinal-audit-jit-run-43451796148.jsonunder/Users/tom/git/Relayer-evidence/issue-555/.Next unit: retain/hash the complete artifact and frozen inventories, independently verify submitted inputs in completed logs/protocol and all four runtime/native identities, then compare every raw listing within and across the baseline bracket and candidates. Retain missing/interleaved/inlined/precompiled methods, repeated tiers and adverse differences. Diagnostic timings and allocation measurements are excluded. Matching listings cannot establish identical execution or historical code; differences cannot establish historical allocation ownership, which tier executed during measurement, or a production defect alone. Preserve every earlier unfavorable outcome and instrumentation/order/drift/loaded-byte-identity limit. This run adds compiled-code evidence, not another allocation-control repetition or a rerun of the strict Performance gate.
No source/user-facing change, docs impact, new per-message cost or unrun-test claim. No owned services. Removing the in-progress claim and releasing issue-555 and its clean isolated analysis worktree. Only this new hosted execution remains pending; no on-hold/needs-human label.
thomhurst commentedon Nov 13, 2026
The earlier wait is resolved: 43451796148 completed successfully. Rechecked and claimed #555 under driver-owned issue-555 (HELD-BY-ME). Fresh origin/main is
0267add; native child/dependency #559 is closed. All fourteen earlier issue-555 branches are ancestors of main; this unit's new analysis branch is unchanged main. No unpublished implementation or open delivering/superseding PR exists.Retained complete artifact
frozen-bdn-control-43451796148-1(14049704707; 92,776,067 compressed bytes) at/Users/tom/git/Relayer-evidence/issue-555/jit-abba/frozen-bdn-control-43451796148-1/. Independently verified all 711 frozen inventory entries and read-back length/SHA-256 verified 844 files / 262,136,662 bytes. All fifteen prior/current retention manifests were also reverified, preserving the original 772-file artifact and every adverse experiment. Reproducer/results:analyze-jit-abba.py,jit-abba-analysis.json/log,jit-abba-retention-sha256.json, all six pairwise raw listing diffs, andjit-prior-inventories.json. Evidence remains local while this issue is open.Actual completed job environment, protocol, native receipts and logs independently confirm Performance 43367252770, concurrent native-parent ABBA, original generated runner 10, eight operations per invoke and runtime 10.0.15 on all four launches. All receipts succeeded with no errors. Hosted launcher tests: 25 passing; no local product tests were needed because no source changed.
All 3,380 raw listing lines / twelve observed method bodies were compared, without discarding repeated tiers or adverse differences. Each arm has Tier0 (677 code bytes), instrumented Tier0 (832), and Tier1 (1358 / 1358 / 1362 / 1360). Tier0 and instrumented Tier0 text match across all four. Candidate 2 Tier1 differs from baseline 1 only in reported PGO count. Candidate 3 reorders resume/completion blocks, changes register selection and bulk-write-barrier call forms. Baseline 4 also differs from baseline 1 in completion-block layout/register selection and direct/indirect bulk-write-barrier calls. PGO counts are 101296 / 199424 / 95856 / 73136. This demonstrates differing observed bodies within the unchanged baseline bracket too; it does not attribute the historical allocation delta or identify an executed tier.
No malformed/interleaved or truncated listing was observed in those twelve bodies. Filtered coverage cannot exclude missing/inlined/precompiled methods or unobserved tiers. All diagnostic timings and allocation values are excluded. Preserve instrumentation, adaptive timing, process order, host drift, historical loaded-byte identity and all previous profiler/protocol confounders. No throughput conclusion, relaxed threshold or accepted trade-off is established.
Resolve audit and finite result: the wait blocker no longer holds. No open prerequisite can recover the original owner events; #559 is closed, and #558/#561/#589/#694 provide tools rather than historical attribution. Main's newer #700 window correlation likewise observes new executions, not missing events from run 43367252770. No already-implemented attribution/fix, delivering/superseding PR, unfinished native child, recoverable branch, or reversible implementation choice supplies the missing history. The first two parent criteria are satisfied; the historical-owner criterion is not. No candidate-specific production defect has been demonstrated, so a speculative patch would not satisfy the correction criterion. Updating stale pending-run text records the completed work; rewriting the historical requirement as a noise claim would be unsupported. No human/upstream/credentials/foreign-lock blocker exists. Another unchanged control cannot resolve this evidence gap; no new run was dispatched.
Leaving #555 open with the original strict failure and explicitly unmet criterion. This is a failed attribution attempt, not a repeated wait or deferral and not a claim of resolution. No source/user-facing change, docs impact or new production per-message cost. No services started. Removing the in-progress claim and releasing issue-555 and the clean isolated analysis worktree.
thomhurst commentedon Nov 13, 2026
Rechecked and claimed #555 under the driver-held issue-555 lock (HELD-BY-ME), against freshly fetched origin/main cb890c0. Sole native child/prerequisite #559 is closed; #561/#558/#589/#607/#610 are on main. All surviving prior issue-555 branches are main ancestors, with no unpublished implementation or open delivering/superseding PR. Reverified all 15 retention inventories, including the original 772 files / 228,297,110 bytes, preserving every unfavorable control. The first two parent criteria are complete; exact historical ownership is not. New execution cannot recreate the original allocation stream, which the aggregate artifact never captured. No candidate-specific production defect is demonstrated.
Resolve audit: no open repository prerequisite to switch to; no already-shipped historical attribution/fix, completed-parent resolution, delivering or superseded PR, duplicate resolution, or recoverable branch. New main receive/cache/pool/failure-policy changes and #705/#717's other-input tools do not attribute this historical run. No human/upstream/credential/foreign-lock gate exists. Updating stale text alone or inventing a production fix would not satisfy the criterion. Linked #537 is now closed through explicit present-day mechanism attribution and durable retention, with historical attribution still unknown: its disposition. That offers an honest, reversible investigation-scope decision rather than repeatedly parking an impossible historical requirement; it does not make #537's evidence attribution for #555.
Distinct route verified: #555's exact-method protocol (43391560343) explicitly says
Native NoAcknowledgementConsoleHost; original anonymous-pipe parent absent. Its later exact same-image controls use that same console parent. The completed native-parent #555 run 43451796148 is a JIT capture whose allocation values are excluded. Main's merged #607/#610 can now capture exactEngine.MeasureWithGcowners with the native BDN anonymous-pipe parent and profiling only in the benchmark child. This removes the parent/profiler perturbation from the new execution's owner question. Inspected actual completed environment logs for all 47 subsequent exact-owner workflow runs; none selected Performance 43367252770, so no existing or competing native-parent owner execution was duplicated.Dispatched exactly one trusted-main execution: 43513326001, explicit
performance_run=43367252770 / parent=native / same_image=comparison / runtime_version=10.0.15. API receipt confirms workflow_dispatch/main, expected.github/workflows/frozen-exact-adaptive-control.ymland head cb890c0. Submitted inputs and API receipt are separate evidence; completed logs/protocol must independently confirm actual execution. Receipts/read-back hash inventory:/Users/tom/git/Relayer-evidence/issue-555/native-parent-exact-{dispatch.json,run-43513326001.json,dispatch-inventory.json}. Existing original/control evidence remains locally retained while open; no permanent-retention claim is made.Next unit must retain/hash the entire artifact and frozen inventories, verify actual inputs/runtime, every native receipt/GC denominator/completion and exact method coverage, then analyze all callback types/counts/sizes and adverse/failed outcomes inside this ABBA. Reuse the native reader; .NET 10.0.15 channel/pooling source confirms native pending-read allocation on singleton/cancellation constraints and handler-box/Action allocation on cache miss/lazy continuation, but source and existing counts are not historical causal evidence. Preserve profiler/JIT/callback/locking, adaptive operation count, process order, drift, historical loaded-runtime-byte identity and GC allocation-context versus object-size limits. No subtraction of independent controls, noise claim, Performance rerun, threshold relaxation, speculative production fix or accepted trade-off.
Updated the body with this bounded present-day mechanism question while retaining the original historical criterion explicitly unmet. Do not start a competing run. After the new evidence, decide and document the investigation scope honestly and preserve all evidence beyond Actions expiry/local cleanup before any closure; do not claim the missing history was recovered.
No source/public behavior change, docs impact or new production per-message cost (0 B added). No local tests needed for dispatch of the reviewed workflow, no unrun tests claimed, no services or additional locks. Removing in-progress and releasing issue-555 and its clean isolated analysis worktree. This is a new execution waiting for completion, not another unchanged deferral.
thomhurst commentedon Nov 13, 2026
Rechecked and claimed #555 under the driver-held issue-555 lock (HELD-BY-ME). The earlier blocker has ended: 43513326001 completed successfully. This completes the bounded present-day mechanism attribution described in the issue's native-parent follow-up, using the transparent scope disposition already applied to linked #537. Exact ownership of the original 749 versus 747/748 B/message result remains unknown: aggregate reports never captured its allocation stream, and executing the images again cannot recover those events. The original strict failure remains unfavorable, not noise or a passing gate.
Acceptance and Resolve audit against fresh origin/main
238c8f2e7025b2a254c0c4d839855de5d3cd279a: sole native child/prerequisite #559 is closed. Main includes #561 same-image controls, #607 native BDN parent (5bdb42a8) and #610 child-only profiling (02fe06f6), plus #694 JIT capture. The first two original retention/analysis criteria are satisfied. The original historical-attribution criterion is explicitly unmet, and no candidate-specific production defect is demonstrated to correct. No open blocker, delivering/superseding PR, recoverable unpublished #555 branch, stale partial implementation that supplies missing history, human decision, upstream release, credential or foreign-lock gate exists. Closing a child alone does not prove historical attribution. The remaining route was to complete the previously recorded bounded question, preserve all evidence independently of cleanup, and explicitly disposition that historical limit, rather than invent a patch or start another unchanged control. This is an investigation-scope resolution, not an assertion that the historical criterion was met.Complete execution validation: all 711 frozen inventory entries and 1,009 artifact files / 264,563,344 bytes independently length/SHA-256 verified. All 16 prior/current retention inventories reverified, including the original 772 files / 228,297,110 bytes. Actual workflow environment and protocol confirm Performance 43367252770, original executable/Default job, BDN 0.15.8, runner 10, eight operations/invoke, original native anonymous-pipe parent, child-only Engine.MeasureWithGc profiling, ABBA and runtime .NET 10.0.15 X64. All receipts have FoundExecutable/IsSuccess true, exit 0, no Errors. Every native GC record agrees with outcome and receipt; every native log has WorkloadResult/AfterAll completion. The four outer .log files are empty because native-parent output is in the separately retained native logs/receipts; those actual outputs were checked. Each exact report has one method entry/normal leave, zero callback/metadata failures, and reconciled total object counts/sizes. No failed or adverse report was dropped.
Both candidates lie inside their baseline bracket here. Timings are diagnostic, not throughput evidence.
Every allocating family: eight stable families contribute exactly 536 object-size B/message in every launch: ServiceProviderEngineScope 48; ConcurrentHandler 24; ConcurrentOrder 24; DI Dictionary 80 and Entry[] 144; Int32 24 and Int32[] 36; Json ArgumentState 56. These types each occur once per operation. The remaining five families account for every other callback:
Pooled handler box and Action counts match exactly in each arm: 50,017 / 12,547 / 49,848 / 12,795. There is no additional background-callback family in this capture; previous captures containing one remain preserved. Complete type names, all raw counts/sizes and normalized values are in
native-parent-exact-analysis.json.The allocating mechanisms build on native features, not a new implementation: .NET 10.0.15 BoundedChannel.ReadAsync allocates pending reads when a cancellable token prevents singleton reuse or the singleton is owned. RentFromCache creates a box on a thread/per-core cache miss, and MoveNextAction lazily creates its continuation. The retained benchmark fixture uses these pooled handlers and channel reads with eight concurrent deliveries; exact fixture/runtime source is archived. Variable rents/read counts explain variation among the counted families in this execution; counts do not provide Action call-site stacks or prove historical scheduling or causation.
Retention beyond local cleanup/Actions expiry: unpublished draft release Performance evidence for #555, ID 444777456, tag
performance-evidence-555, retains all 15,398 original paths / 4,341,552,590 bytes: original unfavorable reports/frozen executables, every adverse control, raw profiles, source, analyses and inventories. Content-addressed ZIP stores identical bytes once, without dropping any path or outcome.manifest.jsonmaps each original path to its length/SHA-256 and stored object; every stored object and original path was independently read-back verified. Archive: 64,119,958 bytes, SHA-256 b91736b720fdf10c3a48d9166de95bc3eaa27aa78f82e8752622640bbcaf71d3; GitHub uploaded asset size/digest matches exactly.complete-evidence-sha256.jsonaccompanies it. Authenticated maintainer access is required. This is draft evidence storage, not a published version/package release, and persists until explicitly deleted; local evidence is disposable.Limits and disposition: ELT/JIT, callback fastpaths/locking, metadata resolution, ReadInitial/ReadFinal and concurrent activity perturb this instrumented execution. Adaptive counts, sequential arms/process order, host drift, IL/metadata substitutions in prior controls and historical loaded-runtime-byte identity remain limitations. Object sizes exclude allocation-context/alignment overhead and are reported separately from native GC bytes. No cross-control subtraction, Performance rerun, threshold relaxation, accepted performance trade-off or candidate-causal/throughput claim is made. No demonstrated production defect remains to correct; replacing native channel/pooling behavior or modifying the fixture/gate to force favorable measurements would be speculative.
Closing the completed bounded mechanism investigation with explicit historical uncertainty and durable preservation of every unfavorable result, following the recorded scope decision. No product source/public behavior change; no docs impact; production per-message cost unchanged, 0 B added. The hosted known-count profiler validation and all completed-method validation steps succeeded. No new local product tests were required for evidence analysis/retention, and none are claimed. No competing workflow, owned service or additional item lock was started. Removing in-progress and releasing issue-555 and its clean isolated analysis worktree.