You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Path.GetRandomFileName() is 7 times slower on Linux compared to Windows #31271
There's some work improving it already in #1433. It should improve performance ever so slightly (as it doesn't always XOR with the result of srand() -- it only does so when it detects that the OS method didn't produce good enough numbers according to a simple heuristic).
Since that function doesn't necessarily needs cryptographically-secure numbers, I can use the current method (that requires a syscall on Linux) just to seed a RNG that's entirely in userspace. Probably something in the family of xorshift, as it requires very little state and is, for many non-cryptographic purposes, good enough. Will do a follow-up patch implementing this.
Ah. GetRandomBytes() isn't supposed to generate cryptographically secure numbers, so maybe GetRandomFileName() shouldn't be using it in the first place?
Since GetRandomBytes() is internal, maybe it could be renamed to GetEntropyForSeeding(), and a PRNG using something from System.Security.Cryptography (AesCcm as it's already implemented, or something like ChaCha20, which is used by OpenSSH-portable to implement arc4random(), or even RandomNumberGenerator) could be used instead?
For places where crypto-safe numbers aren't needed, GetEntropyForSeeding() can be used to kickstart something fast-but-not-crypto-safe.
This way, we clear up this confusion, keep in the managed code for longer, and go down to the native side of things when needed.
The output of Path.GetRandomFileName has 8 bytes in it. I believe that it is misleading to say that it returns cryptographically strong string. It depends on how you are using this string. I think we should remove the cryptographically strong part from the docs.
GetRandomBytes() isn't supposed to generate cryptographically secure numbers
FWIW, here is a discussion how we ended up with this: dotnet/corefx#16652 . The output of it are cryptographically strong bytes (for some definitions of cryptographically strong) on all platforms we care about.
Triage: We should be clearer in the docs around using the terms "cryptographically strong" (probably remove it). Changing the the heuristics here should be done with caution as it will potentially cause undesirable consequences. If we can be confident in the distribution of names, and we don't change the set of characters that are used, than changing this would probably be fine.
usingSystem;usingSystem.IO;usingSystem.Diagnostics;usingBenchmarkDotNet;usingBenchmarkDotNet.Attributes;namespacerand1{[MemoryDiagnoser(false)]publicclassBenchmarks{[Benchmark]publicstringGetRandomFileName()=>Path.GetRandomFileName();[Benchmark]publicstringGetRandomFileName2()=>RandomFileNameGenerator.Get();}filestaticclassRandomFileNameGenerator{privateconstintKeyLength=8;privatestaticReadOnlySpan<byte>Base32Char=>"abcdefghijklmnopqrstuvwxyz012345"u8;privatestaticreadonlyobject_syncState=newobject();publicstaticuint_x=(uint)Environment.TickCount;privatestaticuint_y=842502087;privatestaticuint_z=3579807591;privatestaticuint_w=273326509;publicstaticunsafestringGet(){byte*buffer=stackallocbyte[8];lock(_syncState)NextBytes(buffer);returnstring.Create(12,(IntPtr)buffer,(span,key)=>// 12 == 8 + 1 (for period) + 3Populate83FileNameFromRandomBytes((byte*)key,KeyLength,span));}// implementation of http://vigna.di.unimi.it/ftp/papers/xorshift.pdf, based on George Marsaglia's simple xor-shift pseudo random.// https://github.com/bewaretech/PEA.NET/blob/40f851db/src/PEA/PEA/Util/FastRandom.cs#L149 (Apache 2.0)privatestaticunsafevoidNextBytes(byte*buffer){uintx=_x,y=_y,z=_z,w=_w;inti=0;uintt;for(intbound=KeyLength-3;i<bound;){t=(x^(x<<11));x=y;y=z;z=w;w=(w^(w>>19))^(t^(t>>8));buffer[i++]=(byte)w;buffer[i++]=(byte)(w>>8);buffer[i++]=(byte)(w>>16);buffer[i++]=(byte)(w>>24);}if(i<KeyLength){t=(x^(x<<11));x=y;y=z;z=w;w=(w^(w>>19))^(t^(t>>8));buffer[i++]=(byte)w;if(i<KeyLength){buffer[i++]=(byte)(w>>8);if(i<KeyLength){buffer[i++]=(byte)(w>>16);if(i<KeyLength){buffer[i]=(byte)(w>>24);}}}}_x=x;_y=y;_z=z;_w=w;}// existing helper https://github.com/dotnet/runtime/blob/529804afcd9a84499945467c8d04b21a6f674cbd/src/libraries/System.Private.CoreLib/src/System/IO/Path.cs#L794privatestaticunsafevoidPopulate83FileNameFromRandomBytes(byte*bytes,intbyteCount,Span<char>chars){// This method requires bytes of length 8 and chars of length 12.Debug.Assert(bytes!=null);Debug.Assert(byteCount==8,$"Unexpected {nameof(byteCount)}");Debug.Assert(chars.Length==12,$"Unexpected {nameof(chars)}.Length");byteb0=bytes[0];byteb1=bytes[1];byteb2=bytes[2];byteb3=bytes[3];byteb4=bytes[4];// write to chars[11] first in order to eliminate redundant bounds checkschars[11]=(char)Base32Char[bytes[7]&0x1F];// Consume the 5 Least significant bits of the first 5 byteschars[0]=(char)Base32Char[b0&0x1F];chars[1]=(char)Base32Char[b1&0x1F];chars[2]=(char)Base32Char[b2&0x1F];chars[3]=(char)Base32Char[b3&0x1F];chars[4]=(char)Base32Char[b4&0x1F];// Consume 3 MSB of b0, b1, MSB bits 6, 7 of b3, b4chars[5]=(char)Base32Char[((b0&0xE0)>>5)|((b3&0x60)>>2)];chars[6]=(char)Base32Char[((b1&0xE0)>>5)|((b4&0x60)>>2)];// Consume 3 MSB bits of b2, 1 MSB bit of b3, b4b2>>=5;Debug.Assert((b2&0xF8)==0,"Unexpected set bits");if((b3&0x80)!=0)b2|=0x08;if((b4&0x80)!=0)b2|=0x10;chars[7]=(char)Base32Char[b2];// Set the file extension separatorchars[8]='.';// Consume the 5 Least significant bits of the remaining 3 byteschars[9]=(char)Base32Char[bytes[5]&0x1F];chars[10]=(char)Base32Char[bytes[6]&0x1F];}}}
Note: unix is currently (in main) using non-crypto random and windows using crypto random, while the proposed managed implementation is non-crypto random.
It can be further improved (vectorization, light-weight or no locking etc.), but is the implementation heading in the right direction?
Update: lock-free with `[ThreadStatic]` performing slightly better on osx-arm64:
filestaticclassRandomFileNameGenerator2{privateconstintKeyLength=8;privatestaticReadOnlySpan<byte>Base32Char=>"abcdefghijklmnopqrstuvwxyz012345"u8;[ThreadStatic]privatestaticuint_x;[ThreadStatic]privatestaticuint_y;[ThreadStatic]privatestaticuint_z;[ThreadStatic]privatestaticuint_w;[ThreadStatic]privatestaticbool_isInitialized;privatestaticvoidInitializeState(){if(!_isInitialized){varseed=(uint)Environment.TickCount;_x=seed;_y=seed^842502087;_z=seed^3579807591;_w=seed^273326509;_isInitialized=true;}}publicstaticunsafestringGet(){InitializeState();// Ensure each thread initializes its state oncebyte*buffer=stackallocbyte[KeyLength];NextBytes(buffer);returnstring.Create(12,(IntPtr)buffer,(span,key)=>Populate83FileNameFromRandomBytes((byte*)key,KeyLength,span));}// implementation of http://vigna.di.unimi.it/ftp/papers/xorshift.pdf, based on George Marsaglia's simple xor-shift pseudo random.// https://github.com/bewaretech/PEA.NET/blob/40f851db/src/PEA/PEA/Util/FastRandom.cs#L149 (Apache 2.0)privatestaticunsafevoidNextBytes(byte*buffer){uintx=_x,y=_y,z=_z,w=_w;inti=0;uintt;for(intbound=KeyLength-3;i<bound;){t=(x^(x<<11));x=y;y=z;z=w;w=(w^(w>>19))^(t^(t>>8));buffer[i++]=(byte)w;buffer[i++]=(byte)(w>>8);buffer[i++]=(byte)(w>>16);buffer[i++]=(byte)(w>>24);}if(i<KeyLength){t=(x^(x<<11));x=y;y=z;z=w;w=(w^(w>>19))^(t^(t>>8));buffer[i++]=(byte)w;if(i<KeyLength){buffer[i++]=(byte)(w>>8);if(i<KeyLength){buffer[i++]=(byte)(w>>16);if(i<KeyLength){buffer[i]=(byte)(w>>24);}}}}_x=x;_y=y;_z=z;_w=w;}// existing helper https://github.com/dotnet/runtime/blob/529804afcd9a84499945467c8d04b21a6f674cbd/src/libraries/System.Private.CoreLib/src/System/IO/Path.cs#L794privatestaticunsafevoidPopulate83FileNameFromRandomBytes(byte*bytes,intbyteCount,Span<char>chars){// This method requires bytes of length 8 and chars of length 12.Debug.Assert(bytes!=null);Debug.Assert(byteCount==8,$"Unexpected {nameof(byteCount)}");Debug.Assert(chars.Length==12,$"Unexpected {nameof(chars)}.Length");byteb0=bytes[0];byteb1=bytes[1];byteb2=bytes[2];byteb3=bytes[3];byteb4=bytes[4];// write to chars[11] first in order to eliminate redundant bounds checkschars[11]=(char)Base32Char[bytes[7]&0x1F];// Consume the 5 Least significant bits of the first 5 byteschars[0]=(char)Base32Char[b0&0x1F];chars[1]=(char)Base32Char[b1&0x1F];chars[2]=(char)Base32Char[b2&0x1F];chars[3]=(char)Base32Char[b3&0x1F];chars[4]=(char)Base32Char[b4&0x1F];// Consume 3 MSB of b0, b1, MSB bits 6, 7 of b3, b4chars[5]=(char)Base32Char[((b0&0xE0)>>5)|((b3&0x60)>>2)];chars[6]=(char)Base32Char[((b1&0xE0)>>5)|((b4&0x60)>>2)];// Consume 3 MSB bits of b2, 1 MSB bit of b3, b4b2>>=5;Debug.Assert((b2&0xF8)==0,"Unexpected set bits");if((b3&0x80)!=0)b2|=0x08;if((b4&0x80)!=0)b2|=0x10;chars[7]=(char)Base32Char[b2];// Set the file extension separatorchars[8]='.';// Consume the 5 Least significant bits of the remaining 3 byteschars[9]=(char)Base32Char[bytes[5]&0x1F];chars[10]=(char)Base32Char[bytes[6]&0x1F];}}
Activity
stephentoub commentedon Oct 23, 2019
The bulk of the cost of GetRandomFileName is presumably in getting the randomness, which is also the only part of the code that's platform-specific and differs between platforms.
https://github.com/dotnet/coreclr/blob/master/src/System.Private.CoreLib/shared/Interop/Unix/System.Native/Interop.GetRandomBytes.cs
https://github.com/dotnet/coreclr/blob/master/src/System.Private.CoreLib/shared/Interop/Windows/BCrypt/Interop.BCryptGenRandom.GetRandomBytes.cs
I'm not sure how much that can realistically be improved. Maybe dotnet/corefx#37906 will help.
am11 commentedon Oct 30, 2019
An alternative to OS based random could be managed implementation of http://vigna.di.unimi.it/ftp/papers/xorshift.pdf, based on George Marsaglia's simple xor-shift pseudo random. e.g. https://github.com/bewaretech/PEA.NET/blob/40f851db/src/PEA/PEA/Util/FastRandom.cs#L149, which claims:
AaronRobinsonMSFT commentedon Feb 1, 2020
/cc @lpereira
lpereira commentedon Feb 3, 2020
There's some work improving it already in #1433. It should improve performance ever so slightly (as it doesn't always XOR with the result of
srand()-- it only does so when it detects that the OS method didn't produce good enough numbers according to a simple heuristic).Since that function doesn't necessarily needs cryptographically-secure numbers, I can use the current method (that requires a syscall on Linux) just to seed a RNG that's entirely in userspace. Probably something in the family of xorshift, as it requires very little state and is, for many non-cryptographic purposes, good enough. Will do a follow-up patch implementing this.
stephentoub commentedon Feb 3, 2020
From the docs:
https://docs.microsoft.com/en-us/dotnet/api/system.io.path.getrandomfilename?view=netcore-3.1
"The GetRandomFileName method returns a cryptographically strong, random string that can be used as either a folder name or a file name."
lpereira commentedon Feb 3, 2020
Ah.
GetRandomBytes()isn't supposed to generate cryptographically secure numbers, so maybeGetRandomFileName()shouldn't be using it in the first place?Since
GetRandomBytes()is internal, maybe it could be renamed toGetEntropyForSeeding(), and a PRNG using something fromSystem.Security.Cryptography(AesCcmas it's already implemented, or something likeChaCha20, which is used by OpenSSH-portable to implementarc4random(), or evenRandomNumberGenerator) could be used instead?For places where crypto-safe numbers aren't needed,
GetEntropyForSeeding()can be used to kickstart something fast-but-not-crypto-safe.This way, we clear up this confusion, keep in the managed code for longer, and go down to the native side of things when needed.
stephentoub commentedon Feb 6, 2020
Probably.
jkotas commentedon Feb 6, 2020
The output of
Path.GetRandomFileNamehas 8 bytes in it. I believe that it is misleading to say that it returns cryptographically strong string. It depends on how you are using this string. I think we should remove the cryptographically strong part from the docs.FWIW, here is a discussion how we ended up with this: dotnet/corefx#16652 . The output of it are cryptographically strong bytes (for some definitions of cryptographically strong) on all platforms we care about.
JeremyKuhne commentedon Feb 27, 2020
Triage: We should be clearer in the docs around using the terms "cryptographically strong" (probably remove it). Changing the the heuristics here should be done with caution as it will potentially cause undesirable consequences. If we can be confident in the distribution of names, and we don't change the set of characters that are used, than changing this would probably be fine.
DeerBear commentedon Apr 3, 2020
What's wrong with reading from /dev/urandom as a seed?
am11 commentedon Sep 21, 2024
Benchmark
Note: unix is currently (in
main) using non-crypto random and windows using crypto random, while the proposed managed implementation is non-crypto random.Windows:
// * Summary *
BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4169/23H2/2023Update/SunValley3)
13th Gen Intel Core i7-1365U, 1 CPU, 12 logical and 10 physical cores
.NET SDK 8.0.400
[Host] : .NET 9.0.0 (9.0.24.46307), X64 RyuJIT AVX2
Job-XTXYQT : .NET 9.0.0 (9.0.24.46307), X64 RyuJIT AVX2
Toolchain=.NET 9.0
Linux (same machine using WSL):
// * Summary *
BenchmarkDotNet v0.14.0, Ubuntu 24.04 LTS (Noble Numbat) WSL
13th Gen Intel Core i7-1365U, 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.108
[Host] : .NET 9.0.0 (9.0.24.46307), X64 RyuJIT AVX2
Job-LAVSVF : .NET 9.0.0 (9.0.24.46307), X64 RyuJIT AVX2
Toolchain=.NET 9.0
Mac:
// * Summary *
BenchmarkDotNet v0.14.0, macOS Sonoma 14.6.1 (23G93) [Darwin 23.6.0]
Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
.NET SDK 8.0.401
[Host] : .NET 9.0.0 (9.0.24.46307), Arm64 RyuJIT AdvSIMD
Job-SESMBK : .NET 9.0.0 (9.0.24.46307), Arm64 RyuJIT AdvSIMD
Toolchain=.NET 9.0
It can be further improved (vectorization, light-weight or no locking etc.), but is the implementation heading in the right direction?
Update: lock-free with `[ThreadStatic]` performing slightly better on osx-arm64: