Skip to content

Aggregate RAR search diagnostics in binary logs - #14599

Draft
baronfel wants to merge 2 commits into
mainfrom
baronfel-analyze-binlog-size
Draft

Aggregate RAR search diagnostics in binary logs#14599
baronfel wants to merge 2 commits into
mainfrom
baronfel-analyze-binlog-size

Conversation

@baronfel

@baronfel baronfel commented Jul 31, 2026

Copy link
Copy Markdown
Member

Context

ResolveAssemblyReference (RAR) currently writes one BuildMessageEventArgs for each rejected assembly candidate. Large builds can contain millions of these events.

The analyzed 823.6 MiB binlog contained approximately 19.6 million RAR messages. This change covers 14.7 million messages with 4.45 billion rendered characters.

The projected compressed size is approximately 620 MiB, with a range of 600–650 MiB. This result is a reduction of approximately 175–225 MiB (21–27%).

The estimate includes the affected RAR messages and component-level string deduplication. It excludes RestoreTaskEx, Tsc, and RAR conflict-list output.

Implementation

  • This PR adds AssemblyResolutionSearchTraceEventArgs.
  • RAR writes one unchunked structured search event for each reference.
  • Each event stores the requested identity, search attempts, processor architectures, and message templates.
  • The binary logger deduplicates each component through the string table.
  • The binary logger does not store the rendered multiline message.
  • The event supports standard node transport, task-host transport, and out-of-process RAR nodes.
  • The binary-log format version increases to 28.

Compatibility

Ordinary loggers render one multiline message for each reference. The message contains the same candidate details as the individual messages.

Loggers add per-event timestamps, target names, task identifiers, and similar decorations once for each reference. Previously, loggers added these decorations to each candidate message.

MSBUILDDISABLEFEATURESFROMVERSION=18.11 restores the individual candidate messages. Warning and error behavior does not change.

Older forward-compatible readers skip the new length-prefixed event. These readers do not show the aggregated search trace.

Measured impact

The benchmark uses the actual lazy legacy event shape. It does not format legacy messages during capture or serialization.

Attempts per reference First binlog bytes Steady-state bytes Capture allocation End-to-end time
2 933 → 1,262 72 → 40 696 → 352 B 6.62 → 7.93 μs
32 4,565 → 4,908 568 → 251 5,528 → 2,512 B 23.96 → 24.57 μs
116 15,353 → 15,251 2,220 → 906 21,392 → 8,560 B 75.84 → 67.12 μs
512 66,970 → 65,302 10,367 → 4,635 96,352 → 37,072 B 396.09 → 289.63 μs

The fixed event schema increases the first-event size at 2 and 32 attempts. The size crossover occurs between 32 and 116 attempts.

Capture allocation decreases by 49–62% in all measured cases. At 116 attempts, end-to-end time decreases by 11.5%.

Message rendering remains lazy. If a logger requests the full multiline message, the aggregate path allocates approximately twice as much as the legacy path.

Testing

  • ./build.sh --test -c Release -v quiet -bl --binaryLogName 21.binlog --pipelinesLog
  • ./build.sh -v quiet -bl --binaryLogName 24.binlog --pipelinesLog
  • Focused Build and Tasks unit tests cover serialization and node packets.
  • Focused tests also cover multi-node logging, RAR aggregation, ChangeWave opt-out, and AssemblyFoldersEx behavior.

Design decision

The event does not divide one reference trace into chunks. A reference is the natural unit, and observed chatty references have hundreds of attempts.

Replace per-candidate rejected assembly messages with one structured search trace per reference behind ChangeWave 18.11. Preserve legacy rendering while deduplicating trace components in binary logs and supporting node transport.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b1c8b059-4e8b-4aec-8862-36f474eb2061
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b1c8b059-4e8b-4aec-8862-36f474eb2061
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant