Skip to content

Investigate #537 with exact native adaptive-tail allocation callbacks - #558

Merged
thomhurst merged 5 commits into
mainfrom
issue-537-exact-callbacks
Nov 13, 2026
Merged

thomhurst merged 5 commits into
mainfrom
issue-537-exact-callbacks

Conversation

@thomhurst

@thomhurst thomhurst commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

The completed native-tail sampling control for #537 cannot distinguish exact type counts from weighted allocation ticks. Reuse the existing CLR allocation profiler with synchronous native BDN EventPipe callbacks to retain exact ObjectAllocated notifications and GetObjectSize2 sizes in the final WorkloadActualStop-to-BenchmarkStop envelope, without rebuilding retained workload images or replacing the adaptive engine.

The profiler keeps the complete native event sequence, discards earlier tails after the next timed stop, validates provider/payload/version/thread identity, and stops counting at BenchmarkStop. The existing ABBA launcher now validates every report against native // GC operations, attempts every launch after failure, and preserves every outcome. The new hosted net10 x64 workflow retains original inputs, native SDK/binaries, runtime inventory, reports and failures. Missing/extra/malformed boundaries, metadata failures and inconsistent denominators fail closed.

Native/existing reuse: official pinned EventPipe profiler sample documents synchronous callbacks and provides the SDK/scaffolding; use native ICorProfilerInfo12.EventPipeStartSession, ICorProfilerCallback10.EventPipeEventDelivered and GetCurrentThreadID, plus the existing ObjectAllocated/GetObjectSize2 profiler. BDN 0.15.8 EngineEventSource supplies boundaries; its original runner, adaptive job, waiter, retained-image verifier and ABBA launcher remain in use. Existing allocation-report validation is shared rather than duplicated. No broker capability or engine is reimplemented.

Limits: this envelope includes final timed cleanup/GC/logging and extra-stat setup/work/cleanup, not exact MeasureWithGc. The serialization lock determines boundaries for concurrent callbacks. CLR notifications/object sizes exclude allocation-context overhead; there are no allocation stacks or exact call sites. Callbacks, metadata resolution and locking alter allocation fast paths and scheduling. A new execution cannot attribute the historical owner, establish a production defect, or justify independent-total subtraction, threshold relaxation or a performance trade-off. Preserve the original unfavorable evidence. Related to #537; partial investigation, does not close its historical-owner acceptance criterion.

Validation: native profiler CMake build; fixture Release build with zero warnings/errors; all 12 focused report/native-fixture tests on Linux ARM64 with official .NET 10.0.15, plus all 9 launcher tests; actionlint and diff checks. Known-count cases use 1/1024/16384 objects from four threads, exclude timed work/earlier tails/post-stop work, and reject missing/extra boundaries. Local validation is not retained Linux x64 workload execution: the new PR workflow supplies that separate experiment. Before interpreting it, independently retain/hash every artifact and inspect all type families, native events/denominators, runtime identity and ABBA outcomes within the stated limits.

All 50 source/reference/binary/runtime/report/log files (35,662,393 bytes) were independently copied and read-back length/SHA-256 checked against originals at /Users/tom/git/Relayer-evidence/issue-537/exact-tail-validation-64cd3eddcc6dcb9344fd2f207055ceb018cc0d90/inventory.json, outside removable worktrees while #537 stays open (local retention, not permanent public hosting).

Production per-message cost unchanged, 0 B added. No docs impact (external investigation tooling only); its developer README documents the new mode and limits.

Review follow-up: 9220874 fixes retained uninstrumented launcher replay via allocation-mode-only imports, with a two-script subprocess regression. All 10 focused launcher tests pass; expiry/download prerequisites and DOTNET_ROOT diagnostics are documented. New validation sources/log are independently copied/read-back hash verified at /Users/tom/git/Relayer-evidence/issue-537/exact-tail-review-validation-92208745134f3839017da2ece6efdfa1cd3d11c6/inventory.json; all 50 earlier validation files rehashed successfully. No native profiler/workload behavior changed.

Hosted evidence available to reviewers: successful exact-tail run 43374332036, artifact frozen-exact-tail-43374332036-1 (14013064806; subject to GitHub expiry). Full independent retention and outcome/type analysis covers every launch and all 965 files / 230,423,893 bytes; local retention under issue-537 remains separate from hosted expiry. Native ABBA 729.859863 / 737.056885 / 734.796631 / 735.330566 B/message with all coverage and native denominator checks passing. One candidate above bracket and one within still do not identify historical ownership or establish a production defect.

CI diagnosis follow-up (55ab5165bb3b4d360df4bb73682a4dd5be40820a): preserve the original failing CI run 43375278122, which reports consumer 1,036,360 B versus native 1,024,000 B on both targets. The existing aggregate cannot attribute those extra bytes. The exact test now preallocates storage for every measurement window, records bytes plus generation collection-count deltas outside the measured Run, and formats all 1,000 native/consumer/receive triples only after every measurement completes. Equality, warmup, GC configuration and TUnit parallel execution remain unchanged. Collection deltas can correlate a window with GC activity but do not attribute objects, counter retirement or historical ownership. Instrumentation between calls can change timing; a separate focused execution is not equivalent to the original parallel CI environment.

The new Concurrent allocation window diagnosis workflow runs that single test on hosted Linux net10/net11 without retries and always retains logs, runtime/commit identity, sources and executable inputs for 90 days. It is an independent instrumented capture, not a rerun for green or a substitute for required CI. Its passing result would not explain the original failure. Existing native profiler callbacks remain separate diagnostic evidence, not a claim about these exact unprofiled windows.

Validation: both focused local target runs passed on macOS ARM64 (.NET 10.0.14/11.0.0), all totals 1,024,000 B. All 2,000 window triples are retained in the raw log. Actionlint 1.7.12 and git diff --check pass. Independently copied/read-back length/SHA-256 verified 485 source/binary/report/log files at /Users/tom/git/Relayer-evidence/pr-558/window-diagnostics-55ab516/inventory.json, outside the removable worktree (local retention while this PR stays open, not permanent public hosting).

Reuse: existing ConcurrentReceiveFixture.Run, native GC.CollectionCount, exact allocation assertion, TUnit filter, standard Actions artifact retention. No broker or measurement engine is reimplemented. Production per-message cost unchanged, 0 B; only test diagnostics change. No docs impact (test/investigation tooling). #348's unavailable Windows originals remain a separate evidence gate; this change neither duplicates its held GC mitigation nor establishes it as the cause of #558's larger excess. No production defect or accepted performance trade-off is claimed.

Rebase follow-up: current head 0554cf6377f780a768919395cc5c270002ed2c5b rebases onto main 753247a and resolves shared launcher/test insertion conflicts without dropping main's exact-runtime or same-image controls. The exact-callback mode preserves the same-image limit, and its regression now exercises both ABBA and same-image/exact-runtime cases, including all failure outcomes. All 13 focused launcher tests pass; report validation tests pass (2 Python cases, 2 optional native fixture cases skipped because those environment-configured binaries are not present). No native profiler/workload code or core allocation test code changed during this rebase; their unchanged focused checks were not rerun. Rebase integration source/log inventory is independently copied/read-back length/SHA-256 verified at /Users/tom/git/Relayer-evidence/pr-558/0554cf6377f780a768919395cc5c270002ed2c5b/inventory.json. Earlier diagnostic source/binary retention remains as linked above; its validation was performed before this rebase, not falsely claimed as a new-head binary build.

The merge conflict prevented PR-event captures on the previous head. It is now resolved (MERGEABLE), and focused hosted window diagnosis 43381511914 has started on the current head. Fresh required reviews are requested; remain draft until their current-head verdicts and all findings are dispositioned. Do not merge or infer root-cause resolution from a passing focused capture.

Current integration head 0cf06ab89477a615193627f74d97c72717535d41 rebases onto 3bf00b2 to resolve new launcher import/protocol conflicts. Preserves main's native benchmark selectors/timing parser and this PR's callback validation. Exact callbacks explicitly reject non-concurrent cases before verifying inputs or launching, because their native provider and eight-operation constraints are specific to ConcurrentRoundTrip. Timing parsing is imported after argument parsing so the retained two-script launcher still supports --help. The exact-tail workflow now retains that timing-parser dependency. All 16 launcher tests pass; both Python tail-report tests pass, two environment-dependent native fixture tests skipped. Native profiler and allocation diagnostic test code unchanged; their prior validation is not claimed as rerun. New integration source/log archive (9 files) independently copied/read-back verified: /Users/tom/git/Relayer-evidence/pr-558/rebase-3bf00b2/inventory.json.

Hosted window run 43381511914 completed on prior head 0554cf6: both targets retained 1,000 equal triples, with all byte totals 1,024,000 and zero collection deltas. Independent audit reverified all 2,958 downloaded originals and archived copies against inventory hashes. Earlier local diagnostic archives remain hash-valid, but their removed worktree originals cannot now be independently rechecked; no original-binary recovery is claimed. New-head review/CI still required. This does not explain or replace original unfavorable CI 43375278122 or establish historical allocation ownership. Production per-message cost unchanged, 0 B added; no docs impact.

Current-head new evidence: independently verified 3,924 downloaded original files / 612,550,052 bytes from runs 43383351519 and 43383351422, plus 747 frozen input hashes. Copies and inventories retained under /Users/tom/git/Relayer-evidence/issue-537/ (current-windows-43383351519, current-exact-tail-43383351422, pr-558-current-windows-inventory.json, pr-558-current-exact-tail-inventory.json) while #537 is open. These are new integrated-head experiments, not recovery of removed earlier worktree binaries. Both targets retain 1,000 equal triples and zero GC deltas; exact-tail ABBA 731.035400 / 726.410583 / 726.483643 / 725.840820 B/message, all validation/coverage checks pass. Limits and original unfavorable CI remain unchanged; full audit is in the promotion comment.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1831e613-f470-467b-9a53-f5c0cbfbd3b3
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

Review of #558 (head checked via the diff): I found no blocking issues.

What I checked

  • Native profiler. The nativeTail path in Profiler.cpp fails closed. It validates provider name, event version, payload length, thread identity and the operation count (positive, multiple of 8). Each WorkloadActualStop clears earlier tails, and counting stops at BenchmarkStop. writeReport is only called with gate held (from Shutdown and EndAllocationRegion), so there is no lock re-entry. BeginAllocationRegion and EndAllocationRegion are correctly disabled in tail mode.
  • Python validator. read_native_tail_allocations.py applies the same state machine as the native code (ordering, matched 15/16 pairs, no events after stop). It also checks the denominator against the launcher's native // GC operations. The shared read_rows refactor keeps the existing validator's behaviour.
  • Launcher. Every launch is still attempted after a failure, and the allocation report outcome is recorded per launch. The sampled-trace and callback modes are mutually exclusive.
  • Project rules. This is investigation tooling under tools/ and .github/, with no src/ per-message path change, so no benchmark is needed. It reuses the native EventPipe and ObjectAllocated callbacks rather than re-implementing them, and it adds tests. No user-facing docs are affected.

Non-blocking notes

  1. The workflow hardcodes runtime 10.0.15 and run ID 43124385683. Retained artifacts expire, and the pull_request path trigger will then fail on download rather than on the code under test. Consider restricting the trigger to workflow_dispatch, or recording the expiry in the README.
  2. The runtime inventory step reads DOTNET_ROOT. Fail with an explicit message if it is unset, rather than a bare KeyError.

@thomhurst

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T07:15:47.168696Z 0cf06ab Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64cd3eddcc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/frozen_bdn_control.py Resolved
@thomhurst

Copy link
Copy Markdown
Owner Author

Fixed Codex's blocking retained-launcher replay finding in 9220874: load the allocation reader only when --allocation-profiler is selected. A subprocess regression copies exactly the two retained launcher scripts without either reader and verifies --help; all 10 focused launcher tests pass. Existing allocation-mode tests still validate every report and failure outcome. No native profiler or workload code changed.

Also addressed both nonblocking notes in Claude's review in this blocking-fix push: README documents input expiry/download failure and explicit-run dispatch; workflow gives an actionable unset-DOTNET_ROOT error. No trigger restriction is needed: the original-input prerequisite remains explicit and missing inputs fail closed. git diff --check passed; actionlint is unavailable in this host, so changed workflow syntax is validated by hosted checks. Production per-message cost unchanged, 0 B added; no docs impact beyond the developer README.

Rehashed all 50 original validation files successfully. The four new source files plus focused test log are independently copied/read-back length/SHA-256 verified at /Users/tom/git/Relayer-evidence/issue-537/exact-tail-review-validation-92208745134f3839017da2ece6efdfa1cd3d11c6/inventory.json, outside removable worktrees while #537 is open. This local retention is not permanent public hosting. Remote head verified at 9220874; remains draft for fresh reviews.

@github-actions

Copy link
Copy Markdown

Review of #558

I read the full diff against AGENTS.md and found no actionable issues.

  • Correctness: the native event state machine in Profiler.cpp and the validation in read_native_tail_allocations.py enforce the same ordering.
    • Both reject a duplicate start, an unmatched or mismatched 15/16 pair, any event after BenchmarkStop, a thread change, and a payload that is not a positive multiple of 8.
    • counts.clear() on each WorkloadActualStop correctly leaves only the final tail.
    • failures is never cleared, so a metadata or provider failure always fails the report.
    • writeReport sits in the same (reopened) anonymous namespace as its forward declaration, so there is no linkage problem.
    • ObjectAllocated and the event callback share gate, and GetCurrentThreadID is called outside it, so I see no deadlock.
  • Reuse: the change builds on the existing profiler, the existing read_rows validator and the existing ABBA launcher. It adds no parallel engine. This matches AGENTS.md's search-before-build rule.
  • Launcher: every launch is attempted and every outcome is kept. A missing denominator or report fails closed. The --help test confirms retained uninstrumented artifacts still run without the new readers.
  • Performance and docs: this is external investigation tooling with no production per-message cost. The README documents the new mode and its limits, so "no docs impact" for the website is correct. The workflow uses pinned action SHAs and read-only permissions.
  • Not verified: I did not build the native profiler or run the Python or fixture tests. I reviewed by reading the diff only.

Non-blocking nit: the PR description cites a local /Users/tom/... evidence path that no reviewer can reach. It would help to say where the hosted artifact lives.

@thomhurst

Copy link
Copy Markdown
Owner Author

Completed independent retention/analysis of exact-tail run 43374332036, executed at 64cd3ed. All 965 files / 230,423,893 bytes copied and read-back length/SHA-256 verified under /Users/tom/git/Relayer-evidence/issue-537/exact-tail-43374332036/; adjacent exact-tail-43374332036-retention.json inventories every file and exact-tail-43374332036-analysis.json retains every outcome, native event, type/count/byte family, coverage and installed runtime inventory. All 747 frozen-image entries match original inventories. This is local retention while #537 remains open, not permanent public hosting.

ABBA launch Native operations Native GC B/message Callback objects Callback bytes Type families / events
baseline 1 32,768 729.859863 360,212 23,798,766 45 / 118
candidate 2 32,768 737.056885 362,605 24,034,598 45 / 114
candidate 3 65,536 734.796631 723,555 47,904,310 44 / 100
baseline 4 32,768 735.330566 361,994 23,978,030 45 / 126

Every native sequence passes complete ordering/thread/payload/stop validation; final denominators match native // GC, coverage failures are zero, all types resolve and totals reconcile. Each log names .NET 10.0.15 x64; installed-runtime hashes and native SDK/source/binaries are retained. Installed inventory does not establish historical loaded-module equality. Candidate 2 exceeds its baseline bracket; candidate 3 is within it. Adaptive denominators differ and must stay normalized.

Inspected all 45 distinct type families across all launches. Per-operation DI dictionary entries/dictionaries/scopes (144/80/48 B), JSON ArgumentState (56), ConcurrentOrder/ConcurrentHandler (24 each), int arrays/boxing (36/24) are stable. Scheduling-sensitive families vary: byte-channel blocked reads 121.877930 / 124.391602 / 122.169434 / 121.086914 B/message; handler pooled boxes 48.949707 / 52.116455 / 52.091553 / 53.199707; matching Actions 23.035156 / 24.525391 / 24.513672 / 25.035156. InMemoryMessage blocked reads are 83.988281 / 83.976562 / 83.976562 / 83.997070; int-channel blocked reads 12.008789 / 12.046875 / 12.045410 / 12.011719. Remaining families are BDN/Perfolizer stats, reflection/type metadata, GC information, delegates, strings and arrays; the one absent family in launch 3 is the BDN measurement array. Full raw families/counts are preserved rather than filtered to favorable owners.

These exact notifications separate type counts from earlier weighted ticks, but the envelope includes final timed cleanup/GC/logging and extra-stat setup/work/cleanup, not exact MeasureWithGc. No allocation stacks or exact call sites exist; metadata/locking/callbacks change fast paths and scheduling. Do not subtract callback totals from native GC bytes or compare independent controls as causal deltas. Neither the repeated direction in one candidate nor scheduling-sensitive types identifies the historical owner or establishes a production defect. Original unfavorable evidence remains intact; no threshold relaxation, production fix, accepted trade-off or Closes #537 claim.

Next investigation should isolate the native extra-stat measurement boundary using the existing native engine/profiler facilities, with explicit coverage and instrumentation limits, rather than treating the tail or a fixed-count control as exact measured ownership. #558 remains partial progress. Codex found a genuine replay-dependency bug; fixed and tested in 9220874, with fresh draft reviews requested. No competing experiment or rerun started.

@thomhurst

Copy link
Copy Markdown
Owner Author

@codex review

@thomhurst

Copy link
Copy Markdown
Owner Author

Disposition of the nonblocking hosted-evidence note in Claude review 43374872772: the PR body now directly links run 43374332036 and names downloadable artifact frozen-exact-tail-43374332036-1 (14013064806), separately from independently retained local evidence and its limits. No code push needed. Current head remains draft pending fresh Codex review; no waiting or competing experiment started.

@thomhurst

Copy link
Copy Markdown
Owner Author

Current-head review check: Claude run 43374872772 is CLEAR for 9220874; all review surfaces inspected and the prior replay-dependency finding is fixed/resolved with no newer reply. Codex summary still reports running for this same head, following request #558 (comment) (eyes reaction only). Keep draft until a positive current-head Codex verdict; the request is recent, so no duplicate review or competing workflow was started. No code changes or tests needed in this unit.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 9220874513

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@thomhurst

Copy link
Copy Markdown
Owner Author

Both required reviewers now approve current head 9220874: Claude CLEAR run 43374872772 and Codex completed/no findings in the mutable summary. Rechecked all review bodies, conversation comments and resolved threads; no newer unresolved finding or reply. Two Claude rounds and zero force-pushes remain below the rework budget.

Before promotion, independently rehashed all 965 hosted evidence files and all 55 original/follow-up validation files; copied and read-back verified those plus all nine changed current-head source files into /Users/tom/git/Relayer-evidence/pr-558/92208745134f3839017da2ece6efdfa1cd3d11c6/. inventory.json lists 1,032 archived entries with lengths/SHA-256, including source inventories. Earlier original worktree paths are gone; retained copies, rather than missing originals, were rehashed. Local evidence remains retained while this PR is open; hosted evidence and inference limits remain as linked in the PR body. No code changes or unchanged tests rerun. Proceeding through protected promotion/arming.

@thomhurst
thomhurst marked this pull request as ready for review November 13, 2026 05:56
@thomhurst
thomhurst enabled auto-merge (squash) November 13, 2026 05:56
@thomhurst

Copy link
Copy Markdown
Owner Author

The earlier wait is resolved: 6845395426 was a Codex request comment, not an Actions run. The mutable Codex summary now completes without findings for 9220874, and Claude run 43374872772 is CLEAR. All conversation/review/inline/thread surfaces checked; the replay-dependency thread remains dispositioned with no newer reply.

Current blocker is real failing CI 43375278122, not review delivery. Both net10 and net11 fail ConcurrentReceiveAllocationTests.Pending_cross_thread_receives_allocate_only_native_scopes_after_warmup: native 1,024,000 B versus consumer 1,036,360 B (+12,360). No rerun for green, assertion relaxation or production change. The two-target focused test passes locally (macOS ARM64, runtimes 10.0.14/11.0.0); a separate focused run including PartitionCompletionWindowTests and JsonSerializerTests (GC-collecting neighbours) passes all 72 target/test combinations. Those different executions do not attribute the hosted failure.

Inspected the fixture, consumer CoreOperation and receive State: continuations are context-owned; no demonstrated new per-message defect. #299 already records this fixture's historical +40 B counter excess, and its existing #348 addresses a demonstrated background-GC counter mechanism. This larger +12,360 B failure is NOT established as that same mechanism. Worked linked #348 under its own lock: its required Windows raw attribution archive/originals remain inaccessible, with no mounted Windows volume or matching hosted artifact. Do not duplicate its held change or claim it fixes this new failure without attribution.

Local diagnosis evidence (479 files including fixture/test/project source, both target binary trees and reports) retained at /Users/tom/git/Relayer-evidence/pr-558/ci-diagnosis-9220874/inventory.json; raw failed CI and local logs are under the parent pr-558 directory. Original unfavorable CI is preserved. No superseding PR or already-shipped fix found; #537 remains a separate ongoing investigation, not an acceptance criterion delivered by #558. Next: obtain the #348 original evidence for verification, and/or capture the failing hosted allocation window to attribute this distinct excess. No CI/review on-hold label or unproven blocked_by relationship added.

@thomhurst

Copy link
Copy Markdown
Owner Author

Pushed 55ab516 to make #558's real failing CI actionable rather than repeat an evidence-access deferral. All measurement windows now retain exact bytes and GC-generation observations outside Run; formatting happens after all measurements. Exact equality, warmup, normal GC and parallel execution remain intact. Added a separate focused hosted capture retaining all logs, runtime identity, binaries and sources, with no retries. Local focused net10/net11 tests pass with equal 1,024,000 B totals; actionlint and diff checks pass. Independent 485-file source/binary/report/log inventory is linked in the updated PR body. These different executions do not attribute or replace failed CI 43375278122, which remains preserved.

Rechecked earlier #348 access blocker under its own lock: no mounted Windows volume, no archive in original run 42309147287, no registered self-hosted runner, no matching Windows archive among repository artifacts, and no new access link. Separate macOS evidence cannot verify those Windows originals; no evidence loss is claimed. Resolve alternatives: freshly fetched main lacks the held GC setting, #299 has no native sub-issues and still requires original evidence verification, and no replacement PR or already-delivered implementation exists. Recreating the setting or generating new Windows evidence cannot recover historical originals; no duplicate fix or unsupported dependency was introduced. Fresh main's four new commits concern other native diagnostics, docs and NATS shutdown, not this allocation failure. A blind rerun/rebase would not diagnose it.

The independent capture route avoids treating #348 as a proven prerequisite: its mechanism is not established as the cause of this +12,360 B excess. PR remains draft for fresh current-head review. Pending capture/review is progress after a push, not a credentials deferral or permission to merge. Next unit should inspect all retained windows and runtime identity from the hosted capture; if it passes, do not claim historical attribution or rerun for green. No on-hold label added for this finite workflow/review wait.

@thomhurst

Copy link
Copy Markdown
Owner Author

@codex review

@github-actions

Copy link
Copy Markdown

Review of #558 (exact native-tail allocation callbacks)

This is diagnostic tooling (CI scripts, a native profiler, a workflow and a fixture). It adds no per-message code under src/, so the AGENTS.md performance budget does not apply. The design is sound. It reuses the existing profiler and ABBA launcher, uses native EventPipe synchronous callbacks, and fails closed on missing, extra or malformed boundaries. I read the C++ callback state machine and read_native_tail_allocations.py, and they agree on ordering, thread identity, and the reset at each WorkloadActualStop followed by the stop at BenchmarkStop. I did not build or run anything.

Actionable

  1. Collection counts are captured but never reported (tests/Relayer.Tests/ConcurrentReceiveAllocationTests.cs, Capture and Window). Capture records Gen0/1/2 deltas into Window. The output loop prints only the scoped, consumer and receive byte values, and the header line prints only the byte totals. The comment says the point is to keep every window, but the data a reader would use to tell a GC-induced window from a real one is dropped. Print Gen0/Gen1/Gen2 per window, or remove the counters.

Non-blocking, but worth fixing

  • The workflow hard-codes the runtime version (frozen-exact-tail-control.yml). The runtime inventory step and the log check require 10.0.15, while setup-dotnet installs 10.0.403/404 and 11.0.100. Any runner image or SDK roll that changes the runtime fails the job with a ValueError rather than a measurement result. Derive the expected runtime from the retained // Runtime= lines, or make it an input.
  • The default performance_run input (43124385683) also runs on pull_request. Once the artifact expires, the PR-triggered job fails at gh run download. The README says a download failure is not a workload result, but a red PR check will still look like one. Consider running the PR trigger only for the fixture and unit-test steps.
  • The profiler and the reader hard-code the benchmark name and the % 8 operations invariant. Both are fine for this one control, but the coupling to ConcurrentRoundTrip is implicit. Add a short comment in both places so a later reuse does not fail closed for a confusing reason.
  • The new concurrent-allocation-windows.yml job is named windows but runs on ubuntu-latest. Rename it to avoid confusion with the Windows OS.

No docs impact applies as long as the PR description states it, per AGENTS.md.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 55ab5165bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@thomhurst

Copy link
Copy Markdown
Owner Author

Follow-up correction to the preceding progress note: fresh main's diagnostic changes did create merge conflicts in shared launcher/test insertion points, which prevented the new PR-event capture from starting. Resolved by rebasing onto 753247a and preserving both main's exact-runtime/same-image modes and #558's exact-callback mode. Added combined same-image/exact-runtime/callback failure-path coverage. All 13 focused launcher tests pass; two Python report-validator cases pass, while two environment-dependent native fixture cases are skipped (native code unchanged). Core allocation diagnostic code is unchanged by this rebase, so its already-passing two-target tests were not rerun.

Remote head verified at 0554cf6 and PR is now MERGEABLE. Independent hosted capture is running: https://github.com/thomhurst/Relayer/actions/runs/43381511914 . It retains both target outcomes; no competing manual dispatch or arbitrary rerun started. Fresh Claude review run 43381509610 started automatically and current-head Codex is requested. The body links independently verified current-head integration sources/logs and the earlier 485-file diagnostic retention. Old failed CI 43375278122 remains evidence; #348 evidence access and historical attribution remain unresolved, not bypassed. Keep draft pending these new-head results; this unit pushed concrete diagnostic/conflict fixes rather than deferring unchanged.

@thomhurst

Copy link
Copy Markdown
Owner Author

@codex review

@github-actions

Copy link
Copy Markdown

Review of #558 (exact native-tail allocation callbacks)

This is diagnostic tooling (CI scripts, native profiler, workflow, fixture). It adds no per-message code under src/, so the AGENTS.md performance budget does not apply. The design reuses the existing profiler and ABBA launcher and native EventPipe synchronous callbacks, and the reader fails closed on missing, unordered or malformed boundaries. I read the workflow, read_native_tail_allocations.py and the relevant profiler/test lines; I did not build or run anything.

Actionable (carried over from the previous review, still present at this head)

  1. GC collection counts are captured but never reported. tests/Relayer.Tests/ConcurrentReceiveAllocationTests.cs:54 defines Window(Bytes, Gen0, Gen1, Gen2), but nothing else in the file references Gen0/Gen1/Gen2 or GetCollectionCount. Either print per-window Gen0/1/2 next to the byte values (they are what distinguishes a GC-affected window from a real one), or drop the fields.

Non-blocking

  • frozen-exact-tail-control.yml hard-codes runtime 10.0.15 in the inventory step and the // Runtime= log check, while setup-dotnet installs 10.0.403/404 and 11.0.100. A runner or SDK roll would fail with a ValueError instead of a measurement result. Derive it from the retained // Runtime= lines or make it an input.
  • The default performance_run (43124385683) also applies to the pull_request trigger. After the artifact expires, the PR check goes red at gh run download and looks like a workload failure. Limit the PR trigger to the fixture and unit-test steps.
  • Profiler.cpp:31 (benchmark name) and :153 (% 8), plus the reader's count % 8, hard-code the ConcurrentRoundTrip coupling. A short comment in each place would stop a later reuse failing closed for an unclear reason.

No docs impact applies provided the PR description says so.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 0554cf6377

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@thomhurst

Copy link
Copy Markdown
Owner Author

Disposition of Claude's current-head review 43381509610, and the same finding in 43381225165: incorrect finding, declined with execution evidence. The interpolations at ConcurrentReceiveAllocationTests.cs:36 print each positional Window record struct, whose generated ToString() includes all four properties. Both completed hosted logs explicitly contain scoped=Window { Bytes = 1024, Gen0 = 0, Gen1 = 0, Gen2 = 0 } (and consumer/receive equivalents). All 1,000 triples on each target contain these fields. No captured counters are dropped. Please correct the blocking verdict after checking the retained log; no code churn or rerun for green is needed.

Nonblocking notes disposition: the pinned runtime and retained input are intentional experiment controls; the README already documents expiry/download prerequisite failures and limits. Changing the runtime implicitly would invalidate the control. ConcurrentRoundTrip and its batch-of-eight operation validation are deliberate fail-closed constraints of this named control; broader reuse is outside this PR. These suggestions do not block correctness and are declined without a separate push. The job name windows denotes measurement windows, and both workflow runner and runtime logs identify Linux; renaming is optional. The existing Codex replay-dependency thread remains fixed/resolved, with no newer reply. All review bodies, inline comments, conversation comments and GraphQL threads were inspected.

Completed independent analysis of hosted capture 43381511914, head 0554cf6377f780a768919395cc5c270002ed2c5b: artifacts concurrent-allocation-windows-net10.0-43381511914-1 (14016299841) and concurrent-allocation-windows-net11.0-43381511914-1 (14017090914). Both Linux X64 captures have 1,000 complete triples, zero unequal windows, native/consumer/receive totals each 1,024,000 B, and zero generation-collection deltas in every window. Runtimes identify .NET 10.0.15 and .NET 11.0.0. These focused instrumented passing executions do not explain or replace unfavorable parallel CI 43375278122 or establish historical ownership.

All 2,958 downloaded files / 382,056,618 bytes, including sources, executable inputs, identities, reports and raw logs, were independently copied and read-back length/SHA-256 verified. Inventory and full-window analysis: /Users/tom/git/Relayer-evidence/pr-558/hosted-windows-43381511914-inventory.json; downloaded originals and verified copies are adjacent. Local retention lasts while this PR is open; hosted artifacts have their configured 90-day expiry. No permanent hosting claim.

Earlier #348 blocker was rechecked under its own lock and its existing blocker comment updated: archive/originals remain inaccessible, no matching hosted artifacts or Windows mount/self-hosted runner; main still lacks the setting, #299 has no finished children or stale implementation scope, and no replacement PR exists. Recreating a run cannot verify historical originals. This separate [credentials] gate is not a proven dependency of #558. No new unsupported dependency, duplicate mitigation, on-hold for finite waits, or evidence-loss assertion.

Current-head Codex is completed/no findings. Keep #558 draft pending corrected positive Claude verdict and the protected ready/CI lifecycle; no new review workflow was dispatched to rerun a code finding. No code changed or unchanged tests rerun. Production per-message cost unchanged; no docs impact.

@thomhurst

Copy link
Copy Markdown
Owner Author

Follow-up confirmation request for current head 0554cf6, following the evidence-backed disposition in #558 (comment) . Rechecked every review surface: the sole Codex thread is fixed/resolved with no newer reply; current-head Codex has no findings. Claude's GC-counter finding is incorrect: ConcurrentReceiveAllocationTests.cs:36 interpolates positional Window record structs, whose generated ToString includes Bytes/Gen0/Gen1/Gen2. Both hosted captures retain those fields in all 1,000 triples. No source change or test rerun is needed to manufacture a different result.

Requesting a fresh trusted-main review specifically to confirm or reaffirm that disposition, as draft-review.md step 1 requires when confirmation is needed without a new head. This is not a failed-delivery recovery, rerun of a BLOCKING workflow for green, or permission to ignore a finding. Please inspect the disposition and actual code and issue the current-head verdict; reaffirmed blocking feedback will be handled. No existing review is running and no follow-up dispatch exists for this head. Keep this PR draft until both required positive verdicts and the protected promotion/CI lifecycle. #555 needs this reusable diagnostic tool; do not duplicate it there.

@github-actions

Copy link
Copy Markdown

Review of PR #558 (current head)

No actionable findings.

  • Prior blocking finding (GC counters dropped from output). I checked this against the code and it does not hold. ConcurrentReceiveAllocationTests.cs:36 interpolates the positional Window record struct (line 54). Its generated ToString() prints Bytes, Gen0, Gen1 and Gen2, which matches the scoped=Window { Bytes = 1024, Gen0 = 0, ... } lines the author cites from the hosted logs. The disposition is correct.
  • Profiler.cpp. The native-tail state machine is fail-closed. It validates the provider name, payload length, version, thread identity and operation divisibility. Failures are never cleared, and only the tail since the last timed stop is kept. BeginAllocationRegion and EndAllocationRegion refuse to run in native-tail mode. It reuses the existing ObjectAllocated/GetObjectSize2 profiler and the native EventPipe profiler API rather than re-implementing anything, which fits AGENTS.md's wrap-don't-replace rule.
  • Per-message path. The change is tooling and an investigation test only. Production per-message cost is unchanged and no benchmark is needed. There is no docs impact for the user-facing site.
  • Minor, non-blocking. engineEvents grows with every engine event in the process. That is acceptable for a bounded investigation run, but a cap would make the memory use explicit.

@thomhurst
thomhurst force-pushed the issue-537-exact-callbacks branch from 0554cf6 to 0cf06ab Compare November 13, 2026 07:06
@thomhurst

Copy link
Copy Markdown
Owner Author

Resolved newly developed merge conflicts by rebasing onto main 3bf00b2 and pushed verified remote head 0cf06ab. Preserved main's benchmark selectors/timing output and exact callback controls; reject unsupported callback cases before launch, retain the timing reader in the hosted artifact and keep optional imports out of --help. All 16 focused launcher tests and two Python tail-report tests pass; two native-fixture cases skipped because their configured binaries are unavailable. Native profiler and core allocation test unchanged. Nine integration source/log files independently copied and read-back hash verified; inventory and earlier evidence retention limits are in the updated body. PR is MERGEABLE again.

Disposition of Claude 43382946924's optional engineEvents cap: nonblocking and declined for this bounded investigation control; a separate optional memory-policy change is unnecessary here. Earlier incorrect GC-counter finding was explicitly withdrawn by that CLEAR verdict. All review surfaces were audited; resolved Codex thread has no newer reply. New head requires fresh verdicts; keep draft, no promotion or workflow rerun for green.

#348 is a separate historical evidence-access gate, not a proven prerequisite for this tooling. This push resolves actionable #558 integration work so #555 can reuse it after the normal review/CI lifecycle. No duplicate GC mitigation or allocation threshold changes. No docs impact; production per-message cost unchanged.

@thomhurst

Copy link
Copy Markdown
Owner Author

@codex review

@github-actions

Copy link
Copy Markdown

Review of head 0cf06ab

This PR is investigation tooling only: a profiler native-tail mode, Python report validators and launcher, two workflows, and a diagnostic test. I read the tail validator (read_native_tail_allocations.py) and the PR description and previous review dispositions. I did not build or run the native profiler or the workflows.

  • Correctness: the validator fails closed. It requires exactly one tail record, enforces record ordering, thread identity, benchmark stop, and iteration/operation denominator agreement with the native // GC count. I found no logic errors.
  • Reuse (AGENTS.md): it builds on the existing ObjectAllocated/GetObjectSize2 profiler, native EventPipe session APIs and BDN's EngineEventSource, and shares read_rows instead of duplicating it. This matches wrap-don't-replace.
  • Per-message path: production code is untouched, so the per-message cost is unchanged and no benchmark is needed. There is no docs impact.
  • Prior threads: the earlier GC-counter finding was withdrawn. The optional engineEvents cap was reasonably declined for a bounded investigation run.

No actionable findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 0cf06ab894

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@thomhurst

Copy link
Copy Markdown
Owner Author

Current-head promotion audit (2026-11-13): Claude 43383347764 is CLEAR and Codex's mutable summary is completed/no-findings for 0cf06ab. All review bodies, conversation comments, inline comments and GraphQL threads rechecked; sole inline thread remains resolved with no newer reply. Earlier GC-output finding was withdrawn; optional findings have their recorded dispositions.

Independently downloaded and read-back length/SHA-256 verified all 3,924 files / 612,550,052 bytes against downloaded originals from new hosted runs 43383351519 and 43383351422, plus all 747 frozen input inventory entries. Archived copies and inventories are under /Users/tom/git/Relayer-evidence/issue-537/current-windows-43383351519, current-exact-tail-43383351422, and pr-558-current-{windows,exact-tail}-inventory.json. Retained while #537 remains open, not permanent public hosting; hosted artifacts retain their configured expiry. Windows-window artifacts IDs 14017880953/14016949559 and exact-tail artifact 14017387235. The source identity 5b03f15 is GitHub's merge commit with parents main 3bf00b2 and PR head 0cf06ab; this is integrated head validation, not a different unreported source revision.

Both Linux X64 target captures have 1,000 triples, all native/consumer/receive totals 1,024,000 B, every per-window byte count 1,024 and every generation delta zero. All four exact-tail outcomes validate event/thread/denominator/coverage constraints and .NET 10.0.15 identity, with 43/45/44/44 type families and zero callback failures. Native ABBA totals are 731.035400 / 726.410583 / 726.483643 / 725.840820 B/message; both candidates lie within the baseline bracket. These are separate instrumented experiments, not historical attribution, proof of a production defect, or permission for subtraction/tolerance/performance trade-offs. Original unfavorable CI 43375278122 remains evidence and is not explained by passing focused runs.

All nine integration archive entries rehashed and their seven source files matched the current checkout. Earlier removed worktree originals remain unavailable; these new hosted captures are new evidence, not recovery or independent reverification of those older binaries. Their earlier retention limits remain explicit.

Rechecked #348 under its own lock and updated its existing blocker comment: 857 repository artifacts contain no matching originals; original run has none; no registered self-hosted runner or mounted Windows drive. Main still lacks that setting, #299 has no children and retains its evidence acceptance, and no replacement exists. Every Resolve route remains unavailable for that historical gate; new experiments cannot recover the originals. #348 is not a proven prerequisite for this independently reviewed tooling. No new dependency or duplicate mitigation introduced.

Proceeding through the protected promotion script; required ready CI must pass before merge. No code changed or unchanged tests rerun. Production per-message cost unchanged, 0 B added; no docs impact.

@thomhurst
thomhurst marked this pull request as ready for review November 13, 2026 07:14
@thomhurst
thomhurst enabled auto-merge (squash) November 13, 2026 07:14
@thomhurst

Copy link
Copy Markdown
Owner Author

Protected promotion succeeded for head 0cf06ab: ready and auto-merge armed by Enable-AutoMerge.ps1 after current-head reviews and verified evidence. Required CI run 43384281860 is now in progress: https://github.com/thomhurst/Relayer/actions/runs/43384281860 . No CI wait/rerun or merge bypass; GitHub must require CI Passed before merging. #348 historical access gate remains separate as documented. All unit locks are being released; no owned services were started.

@thomhurst
thomhurst merged commit 9bc20f4 into main Nov 13, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant