[release/7.0] CI failure: RsaCryptRoundtrip_OaepSHA256 CryptographicException "Unknown error"

[carlossanlop]

Found in this unrelated release/7.0 PR: #78250

• Queue: Libraries Test Run release coreclr windows x64 Debug
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=80853&view=logs&j=55b384d0-d6b0-5894-fd8a-d754d713246e&t=8dcacd81-cb5e-5ac9-93e5-c3f19b19e19b&l=105
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-78250-merge-aa9b3b407ec047038c/System.Security.Cryptography.Cng.Tests/1/console.ece8a123.log?helixlogtype=result
• Callstack:

===========================================================================================================

C:\helix\work\workitem\e>"C:\helix\work\correlation\dotnet.exe" exec --runtimeconfig System.Security.Cryptography.Cng.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Cng.Tests.deps.json xunit.console.dll System.Security.Cryptography.Cng.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Security.Cryptography.Cng.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Cng.Tests (found 1049 of 1208 test cases)
  Starting:    System.Security.Cryptography.Cng.Tests (parallel test collections = on, max threads = 2)
    System.Security.Cryptography.Rsa.Tests.KeyGeneration.GenerateMaxKey [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_AllocatingSpan.RsaCryptRoundtrip_OaepSHA256 [FAIL]
      System.Security.Cryptography.CryptographicException : Unknown error (0xc0000001)
      Stack Trace:
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.EncryptDecrypt.cs(278,0): at System.Security.Cryptography.RSACng.TryEncryptOrDecrypt(SafeNCryptKeyHandle key, ReadOnlySpan`1 input, Span`1 output, AsymmetricPaddingMode paddingMode, Void* paddingInfo, Boolean encrypt, Int32& bytesWritten)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.EncryptDecrypt.cs(185,0): at System.Security.Cryptography.RSACng.TryEncryptOrDecrypt(ReadOnlySpan`1 data, Span`1 destination, RSAEncryptionPadding padding, Boolean encrypt, Int32& bytesWritten)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.EncryptDecrypt.cs(33,0): at System.Security.Cryptography.RSACng.TryDecrypt(ReadOnlySpan`1 data, Span`1 destination, RSAEncryptionPadding padding, Int32& bytesWritten)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(231,0): at System.Security.Cryptography.RSA.<Decrypt>g__TryWithDecrypt|16_0(RSA rsa, ReadOnlySpan`1 input, Byte[] destination, RSAEncryptionPadding padding, Int32& bytesWritten)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(1414,0): at System.Security.Cryptography.RSA.TryWithKeyBuffer[TState](ReadOnlySpan`1 input, TState state, TryFunc`1 callback, Boolean tryKeySizeFirst)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(234,0): at System.Security.Cryptography.RSA.Decrypt(ReadOnlySpan`1 data, RSAEncryptionPadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(45,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_AllocatingSpan.Decrypt(RSA rsa, Byte[] data, RSAEncryptionPadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs(334,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.RsaCryptRoundtrip(RSAEncryptionPadding paddingMode, Boolean expectSuccess)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs(308,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.RsaCryptRoundtrip_OaepSHA256()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(69,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
    System.Security.Cryptography.Encryption.TripleDes.Tests.TripleDESContractTests.Windows7DoesNotSupportCFB64 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Encryption.Aes.Tests.AesModeTests.Windows7DoesNotSupportCFB128 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Stream.SignHash_DifferentHashes(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Stream.SignHash_Roundtrip(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Stream.SignHash_TamperedSignature(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Array.SignHash_DifferentHashes(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Array.SignHash_TamperedSignature(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Array.SignHash_Roundtrip(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
  Finished:    System.Security.Cryptography.Cng.Tests
=== TEST EXECUTION SUMMARY ===
   System.Security.Cryptography.Cng.Tests  Total: 2801, Errors: 0, Failed: 1, Skipped: 9, Time: 34.374s
----- end Fri 11/11/2022 23:44:21.07 ----- exit code 1 ----------------------------------------------------------

{
  "ErrorMessage": "CryptographicException : Unknown error (0xc0000001)",
  "BuildRetry": false
}

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Found in this unrelated release/7.0 PR: #78250

• Queue: Libraries Test Run release coreclr windows x64 Debug
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=80853&view=logs&j=55b384d0-d6b0-5894-fd8a-d754d713246e&t=8dcacd81-cb5e-5ac9-93e5-c3f19b19e19b&l=105
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-78250-merge-aa9b3b407ec047038c/System.Security.Cryptography.Cng.Tests/1/console.ece8a123.log?helixlogtype=result
• Callstack:

===========================================================================================================

C:\helix\work\workitem\e>"C:\helix\work\correlation\dotnet.exe" exec --runtimeconfig System.Security.Cryptography.Cng.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.Cng.Tests.deps.json xunit.console.dll System.Security.Cryptography.Cng.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Security.Cryptography.Cng.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Cng.Tests (found 1049 of 1208 test cases)
  Starting:    System.Security.Cryptography.Cng.Tests (parallel test collections = on, max threads = 2)
    System.Security.Cryptography.Rsa.Tests.KeyGeneration.GenerateMaxKey [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_AllocatingSpan.RsaCryptRoundtrip_OaepSHA256 [FAIL]
      System.Security.Cryptography.CryptographicException : Unknown error (0xc0000001)
      Stack Trace:
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.EncryptDecrypt.cs(278,0): at System.Security.Cryptography.RSACng.TryEncryptOrDecrypt(SafeNCryptKeyHandle key, ReadOnlySpan`1 input, Span`1 output, AsymmetricPaddingMode paddingMode, Void* paddingInfo, Boolean encrypt, Int32& bytesWritten)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.EncryptDecrypt.cs(185,0): at System.Security.Cryptography.RSACng.TryEncryptOrDecrypt(ReadOnlySpan`1 data, Span`1 destination, RSAEncryptionPadding padding, Boolean encrypt, Int32& bytesWritten)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSACng.EncryptDecrypt.cs(33,0): at System.Security.Cryptography.RSACng.TryDecrypt(ReadOnlySpan`1 data, Span`1 destination, RSAEncryptionPadding padding, Int32& bytesWritten)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(231,0): at System.Security.Cryptography.RSA.<Decrypt>g__TryWithDecrypt|16_0(RSA rsa, ReadOnlySpan`1 input, Byte[] destination, RSAEncryptionPadding padding, Int32& bytesWritten)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(1414,0): at System.Security.Cryptography.RSA.TryWithKeyBuffer[TState](ReadOnlySpan`1 input, TState state, TryFunc`1 callback, Boolean tryKeySizeFirst)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs(234,0): at System.Security.Cryptography.RSA.Decrypt(ReadOnlySpan`1 data, RSAEncryptionPadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(45,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_AllocatingSpan.Decrypt(RSA rsa, Byte[] data, RSAEncryptionPadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs(334,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.RsaCryptRoundtrip(RSAEncryptionPadding paddingMode, Boolean expectSuccess)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs(308,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.RsaCryptRoundtrip_OaepSHA256()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs(69,0): at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
    System.Security.Cryptography.Encryption.TripleDes.Tests.TripleDESContractTests.Windows7DoesNotSupportCFB64 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Encryption.Aes.Tests.AesModeTests.Windows7DoesNotSupportCFB128 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Stream.SignHash_DifferentHashes(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Stream.SignHash_Roundtrip(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Stream.SignHash_TamperedSignature(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Array.SignHash_DifferentHashes(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Array.SignHash_TamperedSignature(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
    System.Security.Cryptography.EcDsa.Tests.ECDsaTests_Array.SignHash_Roundtrip(ecdsa: ECDsaCng { HashAlgorithm = SHA256, Key = CngKey { Algorithm = ECDSA_P521, AlgorithmGroup = ECDSA, ExportPolicy = AllowPlaintextExport, Handle = SafeNCryptKeyHandle { ... }, IsEphemeral = True, ... }, KeyExchangeAlgorithm = null, KeySize = 521, LegalKeySizes = [KeySizes { ... }, KeySizes { ... }], ... }) [SKIP]
      SignHash not implemented.
  Finished:    System.Security.Cryptography.Cng.Tests
=== TEST EXECUTION SUMMARY ===
   System.Security.Cryptography.Cng.Tests  Total: 2801, Errors: 0, Failed: 1, Skipped: 9, Time: 34.374s
----- end Fri 11/11/2022 23:44:21.07 ----- exit code 1 ----------------------------------------------------------

{
  "ErrorMessage": "CryptographicException : Unknown error (0xc0000001)",
  "BuildRetry": false
}

Author:	carlossanlop
Assignees:	-
Labels:	area-System.Security, test-failure
Milestone:	-

[vcsjones]

Duplicate of #29683