[AR-May]

Fixes #13943

Context

The TaskHost read loop deserialized each incoming packet directly from the pipe stream wrapped in a 1 KB BufferedReadStream. Small primitives were served from that 1 KB buffer, so a real pipe read only happened on a buffer refill (roughly once per 1 KB) - but for a large TaskHostConfiguration (tens of MB of task-item metadata) that still meant thousands of small pipe reads, since each refill requests at most 1 KB.

It also defeated InterningBinaryReader.ReadString's MemoryStream fast path, which reads straight from the backing buffer and decodes the whole string in one shot. That path only fires when the translator's BaseStream is a MemoryStream; backed by BufferedReadStream instead, each string was copied through an intermediate byte buffer and, because a refill returns at most 1 KB, larger strings took the slower multi-chunk decode path with a pooled char buffer.

Changes Made

• Under ChangeWave 18.11, client side communication uses a 64 KB read-ahead buffer and reads each complete packet body into a reusable MemoryStream before deserialization.
• The packet-body buffer is allocated lazily with an initial 128 KB capacity. Buffers larger than 2 MB are released after the packet to avoid retaining unusually large allocations.
• The non-APM parent-node read path also uses 64 KB read-ahead buffering. The .NET Framework APM path remains unchanged.
• BufferedReadStream now supports any Stream and configurable buffer sizes.
• Opting out with MSBUILDDISABLEFEATURESFROMVERSION=18.11 preserves the legacy read paths: TaskHost uses its 1 KB buffer, while the parent-side path reads directly from the pipe. No packet-body buffer is allocated.
• Corrupted extended-header packets now fail through the normal logged link-failure path instead of allowing the exception to escape the read loop.

Testing

Existing unit tests + exp runs + perfstar runs (passed, see comments)

[AR-May]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[AR-May]

publishing report from analyzing the perfstar run.

Perf review of perfstar run for perf/AR-May/fix-task-host-read-from-pipe

Verdict: GOOD — fixes a severe Windows multithreaded (MT) build regression, with no impact on normal (non-MT) builds and one small Linux MT-rebuild item to watch.

Branch = single PerfStar CI run (2026-07-23, Windows = GOLDWIN / Linux = GOLDLIN, all builds passed), compared per scenario against main's recent same-machine baseline (2026-07-16 → 24). Metric is build-time; evaluation-time is used as a control and stays flat in every bucket, so the deltas are real work — not environment drift.

Impact by platform × mode — median build-time Δ (− = faster)

Platform × Mode	scenarios	median Δ	real faster / noise / real slower
Windows · MT	14	−16.7%	7 / 7 / 0
Windows · non-MT	10	−0.7%	0 / 10 / 0
Linux · MT	17	+2.1%	1 / 15 / 1
Linux · non-MT	17	−1.2%	0 / 17 / 0

The entire benefit is concentrated in Windows MT; every non-MT bucket is flat, with zero regressions.

Biggest Windows MT wins

Scenario (Windows, MT)	main (ms)	branch (ms)	Δ
orchard-core warm-restore	107,500	15,523	−86%
orchard-core cold-restore	116,022	24,160	−79%
orchard-core warm-inc-build	169,824	73,645	−57%
orchard-core rebuild	207,471	104,069	−50%
aspire rebuild	80,839	47,212	−42%
aspire warm-inc-build	68,100	43,506	−36%
net8-blazorwasm rebuild	9,125	7,926	−13%

Is MT now the faster mode? (MT vs non-MT, measured in the same run)

Platform	Before (main)	After (fix)
Windows	MT faster in 0 / 7	10 / 11
Linux	12 / 17	18 / 22

Before the fix, MT was slower than single-threaded on 100% of Windows scenarios — down to −456% on orchard-core-warm-restore (107 s vs 19 s). After the fix MT wins almost everywhere. Not universal: xunit-rebuild (Linux) stays MT-slower and widened (−11% → −20%).

Watch item (Linux MT)

orchard-core-rebuild-mt (Linux): 95,970 → 107,396 ms ≈ +11% (~2.9σ above main, above its recent max 90.5k–100.5k). Real for that scenario, but it's a single run — recommend 2–3 reruns to confirm. MT there is still faster than non-MT (+5%).

Caveats

• One run per scenario: large effects are conclusive; < ~5% is within run-to-run noise (small "warm" scenarios have CV up to ~50%; large rebuilds ~2–4%).
• ~40 inc-build / publish / check scenarios have no fresh main baseline (main stopped measuring them on GOLD machines after 2026-06-30) → excluded as inconclusive, not evidence of regression.
• Main's own tiny-project build-times drifted +~75% during July; this fully explains any naive "regression" seen against month-old data.

Bottom line: ship it — the change turns Windows MT from unusable (up to 5.6× slower than single-threaded) into a real ~17% win, leaves normal builds untouched, and only needs a confirmatory rerun on the one Linux MT-rebuild scenario.

[AR-May]

publishing report from analyzing the exp insertion runs.

MSBuild "all tasks out-of-proc" — pipe-read fix vs baseline

Comparison of two experimental VS insertions of the same "all tasks out-of-proc" MSBuild build:

• BASELINE : PR 761729 (branch baseline-all-tasks-in-th, MSBuild 26373.6)
• FIX : PR 761726 (branch fix-task-host-read-from-pipe-all-tasks-in-th, MSBuild 26373.5)
  = baseline + the task-host pipe-read change.

All percentages are the regression relative to the non-OOP main baseline (higher = worse).
Only perf suites ran on both PRs; functional gates (required/optional/experimental tests)
were "notApplicable", so correctness is judged from broken/regressed/fault perf tests.

CORRECTNESS — the fix introduces NO extra breakage vs baseline

• DDRIT (VS64): identical outcome — same 21 regressions + the SAME single broken
  test (UWP64.BuildAndDebug) in both.
• Speedometer/RPS: baseline 28 vs fix 26 confirmed-after-rerun regressions.
  The small deltas are single-metric noise flips in both directions.
• The one fault-event signal (CPlusPlusNoCache.UnrealEngine52 | VS_FaultEvents_devenv)
  appears in BASELINE but NOT in the fix.
• No new broken test, crash, or fault is introduced by the fix.
  => Correctness-wise the fix is equivalent to (slightly better than) baseline; both
  carry the expected all-tasks-OOP risk equally.

PERFORMANCE — Duration / build time: the FIX ROUGHLY HALVES the regression
(Speedometer/RPS, time metrics. Values = extra time over main baseline.)

Test                                                Scenario                                 Baseline            Fix              
--------------------------------------------------  ---------------------------------------  ------------------  -----------------
CPlusPlus.UnrealEngine                              0200.Open Solution                       6,888 ms (93.98%)   3,399 ms (46.37%)
CPlusPlusNoCache.UnrealEngine52                     0200.Open Solution                       10,147 ms (79.84%)  5,271 ms (42.4%) 
OrchardCoreNoCache.TestSolutionOpenClose            0200.Time to Intellisense Without Cache  17,045 ms (36.52%)  7,746 ms (17%)   
OrchardCoreNoCacheResponsive.TestSolutionOpenClose  0200.Time to Intellisense Without Cache  22,561 ms (51.7%)   11,695 ms (28.5%)
PicassoNoCache.TestSolutionOpenClose                0200.Time to Intellisense Without Cache  1,013 ms (26.75%)   1,349 ms (37.58%)
ProjectSystemTest.OrchardCoreBuild                  0100.Rebuild Solution                    29,268 ms (37.84%)  17,217 ms (22.1%)
ProjectSystemTest.OrchardCoreBuild                  0200.Incremental Build                   4,911 ms (53.52%)   3,508 ms (38.9%)

The pipe-read fix cuts the added time from all-tasks-OOP by ~40-55% on every
build / solution-open scenario. Only PicassoNoCache TTI is marginally worse
(~300 ms, within that test's noise).

Metric abbreviations: Procs = OS_ProcessesCreated_Total; CLR_BytesAlloc = CLR_BytesAllocated_NonDevenv;
Duration_Total = Duration_TotalElapsedTime; Duration_Adj = Duration_AdjustedElapsedTime;
VM_Images = VM_AdjustedImagesInMemory_Total; VS_Faults = VS_FaultEvents_Total.

SUMMARY OF FIX IMPACT

• SAFE: no additional breakage vs baseline (identical DDRIT breakage; Speedometer
  diffs are noise; fix shows one fewer fault event and 2 fewer confirmed regressions).
• BIG TIME WIN: on Duration/build scenarios the fix roughly halves the OOP time
  penalty — e.g. UnrealEngine Open Solution ~94%->46% and ~80%->42%; OrchardCore
  Build Rebuild 38%->22%, Incremental 54%->39%; OrchardCore TTI 37%->17% and 52%->28%.
• Allocation / process-count metrics elsewhere are inherent to running tasks OOP and
  are ~unchanged — the fix does not (and is not meant to) address those.
• NET: the task-host pipe-read fix delivers a large, real reduction in the time cost
  of running all tasks out-of-proc, at no correctness cost. The improvement shows in
  the magnitudes, not the regression name-set (both still regress vs the non-OOP main
  baseline, because both run every task out-of-proc).

[Copilot]

Pull request overview

This PR improves out-of-proc TaskHost communications performance by pre-buffering each incoming packet body into a reusable MemoryStream before deserialization, reducing many small pipe reads and enabling InterningBinaryReader.ReadString’s MemoryStream fast-path.

Changes:

• Add a reusable _readBufferMemoryStream to buffer packet bodies and reuse the largest-seen allocation across packets.
• Update RunReadLoop to parse the body length from the 5-byte header, synchronously drain the full body into the MemoryStream, and deserialize from memory (including reading the extended-header version byte from the buffer).
• Remove the BufferedReadStream wrapper and operate on the raw NamedPipeServerStream for both reads and writes.

[AR-May]

exp insertion runs:
https://dev.azure.com/devdiv/DevDiv/_git/VS/pullrequest/761723 - fix
https://dev.azure.com/devdiv/DevDiv/_git/VS/pullrequest/761726 - fix + all tasks oop
https://dev.azure.com/devdiv/DevDiv/_git/VS/pullrequest/761729 - baseline (all tasks oop)

[JanProvaznik]

re internal discussion:
I think the max memory stream should be capped this seems like a memory leak.
And this should be behind a changewave due to node communication changes being super high impact.

But on the positive side I trust the numbers and agree with the direction! This will dramatically fix our -mt story

[github-actions]

🔍 Skill Validator Results

✅ All checks passed

Scope	Checked
Skills	0
Agents	1
Total	1
Severity	Count
---	---:
❌ Errors	0
⚠️ Warnings	0
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	Found 1 agent(s)
ℹ️	Validated 1 agent(s)
ℹ️	✅ All checks passed (1 agent(s))

Full validator output

```text Found 1 agent(s) Validated 1 agent(s) ✅ All checks passed (1 agent(s)) ```

[JanProvaznik]

this looks reasonable, I want a second reviewer (@rainersigwald ?)

pls share the data that lead to the selection of constants

and do not merge until we get restore data.

[AR-May]

MSBuild pipe write-buffer threshold and default sizing

The trim should fire on outliers, not on recurring large packets.
Every recurring packet type has p999 well under 2 MB. Second largest TaskHostConfiguration in MT mode is a bit above 1MB.

The default size should be large enough to absorb the next few packets
without a realloc, small enough to be a genuine memory release compared to maximum size of the buffer.
128 KB covers p95 without issue.

DATA:

Non-MT — per pipe (all packet types combined)

Pipe (roles)	Events	Median	p95	p99	p999	Max
client<->server	584	220	1,950	6,183	77,968	77,968
server<->worker	1,254,768	392	662	4,650	134,629	731,691

MT — per pipe (all packet types combined)

Pipe (roles)	Events	Median	p95	p99	p999	Max
client<->server	450	238	2,482	3,909	77,996	77,996
server<->taskhost	21,496	187	39,832	407,466	709,420	116,689,026

[AR-May]

After collecting more data, unfortunately it seems this pr leads to a regression in non-mt. We need to find a better way to address the issue.
Setup: two copies of .NET 11 P7 daily SDK (11.0.100-preview.7.26377.110).
Baseline = pristine daily. PR14505 = same daily + PR head (fe119d6ca)
MSBuild bits deployed. dotnet build /m (non-mt). OrchardCore (huge sln) +
dotnet new console (tiny csproj). Interleaved per iteration.

Aggregated (10 iters each)

Repo	Scenario	Base mean (ms)	PR mean (ms)	Δ mean %	Base median (ms)	PR median (ms)	Δ median %
OC	inc-cold	65 321	65 956	+0.97 %	64 962	66 090	+1.74 %
OC	inc-warm-2	33 870	34 008	+0.41 %	33 432	33 402	−0.09 %
OC	inc-warm-3	31 905	32 493	+1.84 %	31 854	32 662	+2.54 %
OC	re-cold	148 887	154 487	+3.76 %	147 419	153 779	+4.31 %
OC	re-warm-2	81 366	80 902	−0.57 %	79 261	79 509	+0.31 %
OC	re-warm-3	73 537	74 883	+1.83 %	73 239	74 540	+1.78 %
console	inc-cold	3 656	4 586	+25.45 %	3 854	4 574	+18.67 %
console	inc-warm-2	919	1 038	+12.91 %	919	1 035	+12.56 %
console	inc-warm-3	928	1 018	+9.60 %	924	1 022	+10.59 %
console	re-cold	3 807	4 542	+19.31 %	3 814	4 540	+19.04 %
console	re-warm-2	1 191	1 296	+8.80 %	1 193	1 293	+8.32 %
console	re-warm-3	1 198	1 270	+5.98 %	1 194	1 275	+6.78 %

[AR-May]

I updated the PR, the implementation have been changed. I updated the description accordingly.

I no more see a regression in the console app runs. New PerfStar run shows that we preserve the win: major MT fix for big packets. It also is more-less performance-neutral beyond it. Windows non-MT results are somewhat promising: about 3–5% faster overall, mainly on larger builds, but this is based on a single run. Linux results are neutral, with no clear regressions.