ConnectWithRevocation_WithCallback(checkRevocation: True) failed on macOS 12 with RevocationStatusUnknown

[danmoseley]

Affected tests:

• ConnectWithRevocation_WithCallback in last 120 days - System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback
  • Frequency as of 8/25: 12x runs affected (all OSX.1200.Amd64.Open on Mono and CoreCLR)
    • first occurrence on 8/15 - PR 1945162
    • last occurrence on 8/23 - the test was disabled in main (8.0) in PR Disable ConnectWithRevocation_WithCallback test #74168 and in 7.0 (RC2) in PR [release/7.0] Disable ConnectWithRevocation_WithCallback test #74483
• ConnectWithRevocation_ServerCertWithoutContext_NoStapledOcsp in last 120 days - System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_ServerCertWithoutContext_NoStapledOcsp

Frequency: ConnectWithRevocation_ServerCertWithoutContext_NoStapledOcsp

• Error: Assert.Contains() Failure Not found: RevocationStatusUnknown In value: X509ChainStatusFlags[] [Revoked]

Day	Run	Details
8/24	Rolling run 1964517 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open
8/23	Rolling run 1963307 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open
8/19	Rolling run 1955773 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open

Note it retried the job, so it failed twice in a row
https://dev.azure.com/dnceng/public/_build/results?buildId=1947340&view=ms.vss-test-web.build-test-results-tab&runId=50144598&resultId=201890&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Console log: 'System.Net.Security.Tests' from job afed2094-0e40-4bd4-a0ee-55fa1b2cc3d1 workitem b1b033f0-1b26-438c-bff4-e307225d6774 (osx.1200.amd64.open) executed on machine dci-mac-build-316.local running macOS-10.16-x86_64-i386-64bit
...
Error message
Assert.Contains() Failure
Not found: Revoked
In value:  X509ChainStatusFlags[] [RevocationStatusUnknown]


Stack trace
   at System.Net.Security.Tests.CertificateValidationRemoteServer.<ConnectWithRevocation_WithCallback_Core>g__CertificateValidationCallback|6_0(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 351
   at System.Net.Security.SslStream.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs:line 1031
   at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 506
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 519
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 332
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 90
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 308
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
--- End of stack trace from previous location ---

failed in #74023 (comment)

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Note it retried the job, so it failed twice in a row
https://dev.azure.com/dnceng/public/_build/results?buildId=1947340&view=ms.vss-test-web.build-test-results-tab&runId=50144598&resultId=201890&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Console log: 'System.Net.Security.Tests' from job afed2094-0e40-4bd4-a0ee-55fa1b2cc3d1 workitem b1b033f0-1b26-438c-bff4-e307225d6774 (osx.1200.amd64.open) executed on machine dci-mac-build-316.local running macOS-10.16-x86_64-i386-64bit
...
Error message
Assert.Contains() Failure
Not found: Revoked
In value:  X509ChainStatusFlags[] [RevocationStatusUnknown]


Stack trace
   at System.Net.Security.Tests.CertificateValidationRemoteServer.<ConnectWithRevocation_WithCallback_Core>g__CertificateValidationCallback|6_0(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 351
   at System.Net.Security.SslStream.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs:line 1031
   at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 506
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 519
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 332
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 90
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 55
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 308
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
   at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs:line 309
--- End of stack trace from previous location ---

Author:	danmoseley
Assignees:	-
Labels:	area-System.Net.Security
Milestone:	-

[carlossanlop]

Hit it here too:

Libraries Test Run release mono OSX x64 Debug https://github.com/dotnet/runtime/pull/73985/checks?check_run_id=7865910786

[carlossanlop]

@dotnet/ncl this failure is fairly recent, and I'm seeing it in the RC1 branch. Something to keep in mind in case it needs a backport.

[wfurt]

added in #69833. cc @bartonjs. We may put in some debug to the AIA HttpListeners....

[carlossanlop]

One more repro, this time in main: #74101

    System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback(checkRevocation: True) [FAIL]
      Assert.Contains() Failure
      Not found: Revoked
      In value:  X509ChainStatusFlags[] [RevocationStatusUnknown]
      Stack Trace:
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs(351,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.<ConnectWithRevocation_WithCallback_Core>g__CertificateValidationCallback|6_0(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Protocol.cs(1031,0): at System.Net.Security.SslStream.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs(506,0): at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs(519,0): at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
        /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs(332,0): at System.Net.Security.SslStream.<ForceAuthenticationAsync>d__146`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(90,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(55,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs(308,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs(309,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs(309,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs(309,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext)
        /_/src/libraries/System.Net.Security/tests/FunctionalTests/CertificateValidationRemoteServer.cs(309,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.ConnectWithRevocation_WithCallback_Core(X509RevocationMode revocationMode, Nullable`1 offlineContext)

[CarnaViire]

Triage: We should look at it in 8.0. @rzikm to check how much this is failing, and we should disable the test if it hurts CI (and backport disabling the test to 7.0)

[carlossanlop]

@CarnaViire does this meet the bar for 7.0? It was initially found in an RC1 backport PR: #74023

[akoeplinger]

This also happened on a rolling build of release/7.0-rc1: https://dev.azure.com/dnceng/public/_build/results?buildId=1948523&view=results

[rzikm]

Fails quite often (4/day)

2022-08-18T02:03:00.016Z
2022-08-17T23:16:43.35Z
2022-08-17T22:43:35.342Z
2022-08-17T13:45:15.352Z
2022-08-17T06:18:06.761Z
2022-08-16T21:29:30.516Z
2022-08-16T20:59:02.283Z
2022-08-16T20:27:38.852Z
2022-08-16T20:12:48.139Z
2022-08-16T19:40:53.34Z
2022-08-15T23:31:56.85Z

I think we should disable this test on unixes until we address this.