Repository navigation
Test failure: System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests/CreateChain_Hybrid #25979
Copy link
Copy link
Closed
Labels
Known Build ErrorUse this to report build issues in the .NET Helix tabUse this to report build issues in the .NET Helix tabarea-System.Securitytracking-external-issueThe issue is caused by external problem (e.g. OS) - nothing we can do to fix it directlyThe issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Description
{
"ErrorMessage": "Chain is valid on execution 1",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}The test System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests/CreateChain_Hybrid has failed.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException : An internal consistency check failed
Stack Trace:
at System.Security.Cryptography.CngKeyLite.GenerateNewExportableKey(String algorithm, Int32 keySize) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.Algorithms\src\System\Security\Cryptography\CngKeyLite.cs:line 138
at System.Security.Cryptography.RSAImplementation.RSACng.GetDuplicatedKeyHandle() in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.Algorithms\src\System\Security\Cryptography\RSACng.cs:line 34
at System.Security.Cryptography.RSAImplementation.RSACng.ExportKeyBlob(Boolean includePrivateParameters) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.Algorithms\src\System\Security\Cryptography\RSACng.cs:line 48
at System.Security.Cryptography.RSAImplementation.RSACng.ExportParameters(Boolean includePrivateParameters) in E:\A\_work\2\s\corefx\src\Common\src\System\Security\Cryptography\RSACng.ImportExport.cs:line 118
at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey(RSA rsa) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\src\System\Security\Cryptography\X509Certificates\RSAPkcs1X509SignatureGenerator.cs:line 33
at System.Security.Cryptography.X509Certificates.RSAPkcs1X509SignatureGenerator.BuildPublicKey() in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\src\System\Security\Cryptography\X509Certificates\RSAPkcs1X509SignatureGenerator.cs:line 28
at System.Security.Cryptography.X509Certificates.CertificateRequest..ctor(String subjectName, RSA key, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\src\System\Security\Cryptography\X509Certificates\CertificateRequest.cs:line 136
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.OpenCertRequest(String dn, AsymmetricAlgorithm key, HashAlgorithmName hashAlgorithm) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\tests\CertificateCreation\CertificateRequestChainTests.cs:line 210
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateChainRequest(String dn, AsymmetricAlgorithm key, HashAlgorithmName hashAlgorithm, Boolean isCa, Nullable`1 pathLen) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\tests\CertificateCreation\CertificateRequestChainTests.cs:line 252
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateAndTestChain(AsymmetricAlgorithm rootPrivKey, AsymmetricAlgorithm intermed1PrivKey, AsymmetricAlgorithm intermed2PrivKey, AsymmetricAlgorithm leafPubKey) in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\tests\CertificateCreation\CertificateRequestChainTests.cs:line 361
at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateChain_Hybrid() in E:\A\_work\2\s\corefx\src\System.Security.Cryptography.X509Certificates\tests\CertificateCreation\CertificateRequestChainTests.cs:line 60
Build : Master - 20180423.04 (Core Tests)
Failing configurations:
- Windows.81.Amd64-x86
- Release
Runfo Tracking Issue: buildinvalidsignaturetwice
| Build | Definition | Kind | Run Name |
|---|---|---|---|
| 267807 | runtime | PR 84864 | net8.0-linux-Debug-arm64-Mono_release-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8 |
| 267679 | runtime | PR 85858 | net8.0-linux-Release-arm64-CoreCLR_checked-(Alpine.315.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.15-helix-arm64v8 |
| 267024 | runtime | PR 85966 | net8.0-linux-Debug-arm64-Mono_release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-helix-arm64v8 |
| 266120 | runtime | PR 84864 | net8.0-linux-Release-arm64-CoreCLR_checked-(Alpine.315.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.15-helix-arm64v8 |
| 247295 | runtime | PR 85151 | net8.0-linux-Debug-x64-CoreCLR_checked-(Alpine.315.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.15-helix-amd64 |
| 240573 | runtime | PR 84824 | net8.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open |
| 233949 | runtime | PR 84591 | net8.0-windows-Debug-x86-CoreCLR_checked-Windows.10.Amd64.Open |
Build Result Summary
| Day Hit Count | Week Hit Count | Month Hit Count |
|---|---|---|
| 3 | 4 | 7 |
Known Issue Error Message
Fill the error message using known issues guidance.
{
"ErrorMessage": "",
"BuildRetry": false
}Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Known Build ErrorUse this to report build issues in the .NET Helix tabUse this to report build issues in the .NET Helix tabarea-System.Securitytracking-external-issueThe issue is caused by external problem (e.g. OS) - nothing we can do to fix it directlyThe issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Activity
danmoseley commentedon Apr 24, 2018
Not much to go on but we can start tracking ..
bartonjs commentedon Apr 25, 2018
NTE_INTERNAL_ERRORfrom asking CNG to generate an RSA key. Without a stable repro it'll be hard to do anything. We can leave this open for a little bit to see if maybe we have a bad machine and can track down what this particular one means.But NTE_INTERNAL_ERROR is the moral equivalent of Debug.Assert. A late check says that something previously went wrong and didn't notice, and now it doesn't know what went wrong or why, just that hum was not the right answer.
ViktorHofer commentedon May 29, 2019
Same cause on a different test
System.Security.Cryptography.Dsa.Tests.DSASignVerify_Stream/InvalidKeySize_DoesNotInvalidateKey.https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F37451~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190529.12/workItem/System.Security.Cryptography.Cng.Tests/analysis/xunit/System.Security.Cryptography.Dsa.Tests.DSASignVerify_Stream~2FInvalidKeySize_DoesNotInvalidateKey
What can we about this kind of flakiness? It happened on the same configuration, thus should we disable it for Win81?
bartonjs commentedon May 29, 2019
I don't think we can disable all of crypto on Win8.1 😄.
I've reached out to Windows CNG to see if they have any advice.
ViktorHofer commentedon May 30, 2019
Happened again on Win81: https://dev.azure.com/dnceng/public/_build/results?buildId=204333&view=ms.vss-test-web.build-test-results-tab&runId=4950354&resultId=498464&paneView=debug
bartonjs commentedon May 30, 2019
Interesting that it's the same test as before. While sending data to Windows I noticed that the second ever report of this error was for RSA (not DSA) and on Windows 10. The first was DSA, Windows version unknown, a different test.
If there's a semi-reliable way to force this to happen I'd love to send a repro to Windows and let them catch it in action.
ViktorHofer commentedon May 30, 2019
I would try to run the test on a Windows 81 machine in a loop (200-500 times should suffice).
ViktorHofer commentedon May 30, 2019
Failed again but this time in System.Net.Security.Tests:
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38021~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190530.12/workItem/System.Net.Security.Tests/wilogs
bartonjs commentedon May 31, 2019
Windows suggests that this error code mostly means there's a disk problem (e.g. disk full). If the frequency has started to pick up on this then perhaps our Windows 8.1 machines/images need some sort of cleanup.
ViktorHofer commentedon May 31, 2019
@MattGal disk space problems on win81 (see above). Can you please verify?
bartonjs commentedon Jun 3, 2019
Even if the disk/drive as a whole is fine, it may be that the CNG key directories are having issues (https://docs.microsoft.com/en-us/windows/desktop/seccng/key-storage-and-retrieval#key-directories-and-files).
The most likely culprit being
%APPDATA%\Microsoft\Crypto\Keysfor the user the test runs as.ViktorHofer commentedon Jun 3, 2019
What's your recommendation? Move all tests to Outerloop for that one configuration? Or could there be a way to diagnose the health of the machine?
bartonjs commentedon Jun 3, 2019
It would literally mean every crypto test and every System.Net.Security test. So, I don't think that would be a good strategy (particularly if the problem is environmental).
I'm open to suggestion. Adding a cctor which prints the size on disk remaining and number of files in
%APPDATA%\Microsoft\Crypto\Keysuntil we get a hit is maybe reasonable, but since we've seen it from at least three different test assemblies it feels like maybe playing whack-a-mole. Printing it for absolutely every test library ever seems extreme (but with good coverage).Can we inject a diagnostic
dircommand into RunTests.bat from corefx which we can see via MC for the helix job console capture?MattGal commentedon Jun 3, 2019
@bartonjs / @ViktorHofer Unfortunately the machine(s) that ran your tests have long since been recycled, but I am very skeptical of the "disk full" theory going on here. The Windows 8.1 VMs get created with 100+ GB free disk space on the C:\ drive where %APPDATA% is located, refuse to take work if there's less than 3 GB available on this disk, and get deleted whenever there isn't active work to be done.
Have you actually run this on a Windows 8.1 / Server 2K12R2 machine? It might be an actual problem, though it could be caused by other work items messing with the machine state too.
bartonjs commentedon Jun 3, 2019
@MattGal Yep, my main workstation is 2K12R2. I've never seen NTE_INTERNAL_ERROR on it.
15 remaining items
bartonjs commentedon Jun 15, 2022
We added a retry a long while ago, I don't believe it helped. The underlying OS team's response, essentially, was that if we reimaged our test machines more often we'd probably never see the problems (and that they've never seen these sorts of errors on normal user machines).
MattGal commentedon Jun 15, 2022
For every queue listed in the above query we stay up to date with the most recent Azure gallery images for Server 2K12 R2 and 2016, so the images aren't too old, and most of these machines only live a few hours (pick any work item where this failed and I can demonstrate this via Kusto query). Is it possible that the problem isn't "we don't reimage often enough"?
karelz commentedon Jun 24, 2022
No hit in last 10 days (Runfo says 3 hits per month), removing blocking-clean-ci label.
stephentoub commentedon Jul 22, 2022
Just failed here on Windows 10:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72657-merge-06ecc243922e4a95a0/System.Security.Cryptography.Cng.Tests/1/console.28d88923.log?helixlogtype=result
bartonjs commentedon Jul 22, 2022
There are two things we can do here that might help
Switch from NCrypt.dll to BCrypt.dll for weakly typed ephemeral keys (e.g. RSA.Create()) (a thing the OS team has suggested we do in the past, for unrelated reasons). This will cut down on NCrypt <-> LSASS communication and might help here, at the cost of more code and more tests (but better perf and a big shrug on security).
Make the test libraries use pooled keys, to cut down on object registration and deregistration in the LSA bridge.
The first isn't for 7; the second we can try once higher priority issues run out (which may or may not still be during 7).
karelz commentedon Sep 14, 2022
New hit in release/7.0 branch - 9/12 Rolling run 13309:
Platform net7.0-windows-Release-x64-NativeAOT_Release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open:
karelz commentedon Sep 15, 2022
Another hit on different test in release/7.0 branch - 9/13 Rolling run 16085:
Platform: net7.0-windows-Release-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open
Console log:
karelz commentedon Sep 15, 2022
@bartonjs not sure if we were just lucky to get the failure 2x recently, or if it is more common now from some reason.
karelz commentedon Sep 23, 2022
Another hit in release/7.0-rc2 branch - 9/22 Rolling run 26747:
Platform: net7.0-windows-Release-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open
Console log:
steveisok commentedon Feb 28, 2023
@jeffhandley this seems to be happening quite a bit lately seemingly on the mono configurations. Can you have someone take a look?