Data Corruption in System.Buffers.Text.Base64.EncodeToUtf8

@eiriktsarpalis

We started observing intermittent CI failures on tests involving Base64 encoding in late June. This is recognized as a high-risk potential regression lurking in the .NET 10 release, and we need to find the root cause of this in time to resolve it before the .NET 10 RC1 release.

Here are the CI failures associated with this root cause:

Both #117163 and #117169 were reported on June 30, which is the earliest recognized signal of this issue. As was noted in #117665:

That seems... bad?

-f6 ba 82 83 26 0c 39 91 1b 8b 5d b1 8a 0f f3 6a 78 d5 59 a8 0d 64 29 3d d0 0c 35 87 56 00 ...
+f4 0a 80 80 06 00 38 01 00 88 0d 80 88 0f c0 68 08 c0 58 08 00 64 09 00 d0 0c 00 84 06 00 ...
It's like something is clearing every other 6 bits?

F4BA XOR F40A is 02B0: 0b0000_0010_1011_0000

8B5D XOR 880D is 0350: 0b0000_0011_0101_0000

Since base64 is a 6-bit map... I wonder if this is a base64 encoding problem.

This is happening across multiple architectures and operating systems, as well as across different physical machines.

@eiriktsarpalis conducted a preliminary investigation, reporting:

The failing tests seem to all share a call to System.Buffers.Text.Base64.EncodeToUtf8, with that method producing corrupted data

Some of the tests exercise the API through the System.Convert APIs, which only use the Base64 class implementation if vectorization is available and applicable

runtime/src/libraries/System.Private.CoreLib/src/System/Convert.cs

Lines 2451 to 2454 in 9df2042

    
           if (Vector128.IsHardwareAccelerated && !insertLineBreaks && bytes.Length >= Base64VectorizationLengthThreshold) 
        
           { 
        
               ToBase64CharsLargeNoLineBreaks(bytes, chars, charLengthRequired); 
        
           }

The relevant implementation in System/Buffers/Text/Base64Helper/Base64EncoderHelper.cs hasn't been updated since October 2024

/cc @artl93 @jeffschwMSFT @marek-safar @lewing for visibility/awareness

Tagging subscribers to this area: @dotnet/area-system-buffers
See info in area-owners.md if you want to be subscribed.

With the issue ostensibly narrowed down to System.Buffers.Text.Base64.EncodeToUtf8, a potential next step is to write a fuzz test against this method with instrumentation that would capture a memory dump when the test fails. Other instrumentation within the failing tests might also yield clues.

@tannergooding

@tannergooding also suggested that once we can repro this, we should run the repro with:

DOTNET_EnableAVX512=0 to disable vectorization (edited to reflect this is case-sensitive on non-Windows platforms)

If we narrow it down to a vectorization issue, we can use the following to narrow it down further with AVX512 enabled:

DOTNET_ EnableEmbeddedBroadcast=0
DOTNET_ EnableEmbeddedMasking=0

DOTNET_EnableAvx512=0 to disable vectorization

A note: it's DOTNET_EnableAVX512=0 since it's case sensitive on non-windows platforms

I've reviewed all of the failures and have only found occurrences on Amd64 hardware. There was one that appeared to be Arm64, but it looks to be a false-positive match on the failure.

I don't know if it's useful for differential diagnosis, but in case it is, I've never seen a repro on this CPU:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 2
model name      : AMD FX(tm)-8350 Eight-Core Processor
stepping        : 0
microcode       : 0x6000852
cpu MHz         : 1400.000
cache size      : 2048 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 16
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs            : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed ibpb_no_ret
bogomips        : 8035.70
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro

Interesting, seems to be some old CPU without AVX2 (AVX only).

@EgorBo

@EgorBo Might be old, but it does what I need it to 😄. Since there seems to be speculation that it has to do with JIT optimizations and/or intrinsics-enhanced codepaths, I figured I'd share a case where the corruption hasn't ever appeared.

I tried to optimistically repro it by comparing Base64.EncodeToUtf8 output with a naive version generated by GPT for random inputs - was not able to reproduce any mismatch (also tested various EnableAVX knobs). https://gist.github.com/EgorBo/1bf17a02c94d587917d81789d53af847

Probably also makes sense to try it on inputs created with VirtualAlloc to detect cases when it reads out of bounds (if it does).

I think we need a dump and/or more info about input size it failed on

Hm.. probably found something, this repro fails on Checked runtime built from source (main) with:

Unhandled exception. System.Exception: Mismatch in Base64 encoding!
   at Program.<>c.<Main>b__0_0(Object _) in C:\Users\egorb\source\repos\ConsoleApp19\ConsoleApp19\Program.cs:line 34
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

but only with AVX512

@JulieLeeMSFT

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Doesn't repro with DOTNET_TieredCompilation=0 or DOTNET_TieredPGO=0.

This points to a JIT bug. cc @dotnet/jit-contrib This is .NET 10 blocker.

@tannergooding

Bisected to #116983 cc @tannergooding

It's looking like this is hit during Tier1 when we rejit Test:

; Assembly listing for method Program:Test() (Tier1)
; Emitting BLENDED_CODE for generic X64 + VEX + EVEX on Windows
; Tier1 code
; optimized code
; optimized using Synthesized PGO
; rbp based frame
; fully interruptible
; with Synthesized PGO: fgCalledCount is 26.9966
; 30 inlinees with PGO data; 73 single block inlinees; 11 inlinees without PGO data

Enabling MinOpts for this method in particular causes it to no longer repro. It appears as though we're inlining a SequenceEquals and the Base64Helper.EncodeTo call, where we previously didn't. The latter is what is causing the problem, in particular the inlining of Avx512Encode and Avx2Encode simultaneously causes the issue.

That indicates this might be VN or CSE related.

Not morph or constant folding related.

~~Issue is somewhere in the handling of CompareGreaterThanMask for TYP_SIMD32~~

Actually nevermind, that alone isn't sufficient for catching the issue.

Explanation of the issue and the fix is here: #118154

Navigation Menu

Repository navigation

Data Corruption in System.Buffers.Text.Base64.EncodeToUtf8 #118143

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (Vector128.IsHardwareAccelerated && !insertLineBreaks && bytes.Length >= Base64VectorizationLengthThreshold)
	{
	ToBase64CharsLargeNoLineBreaks(bytes, chars, charLengthRequired);
	}

Data Corruption in System.Buffers.Text.Base64.EncodeToUtf8 #118143

Description

Activity

dotnet-policy-service commented on Jul 28, 2025

jeffhandley commented on Jul 28, 2025

jeffhandley commented on Jul 28, 2025

EgorBo commented on Jul 28, 2025

jeffhandley commented on Jul 28, 2025

bartonjs commented on Jul 28, 2025

EgorBo commented on Jul 28, 2025

bartonjs commented on Jul 28, 2025

EgorBo commented on Jul 28, 2025

EgorBo commented on Jul 28, 2025

EgorBo commented on Jul 29, 2025

1 remaining item

dotnet-policy-service commented on Jul 29, 2025

jkotas commented on Jul 29, 2025

EgorBo commented on Jul 29, 2025

tannergooding commented on Jul 29, 2025

tannergooding commented on Jul 29, 2025

tannergooding commented on Jul 29, 2025

tannergooding commented on Jul 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions