Repository navigation
Description
Summary
This is a concrete proposal for a flaky test management system that can be provided by test platform with a cooperation with test frameworks.
Goals:
- Have a native experience for managing flaky tests.
- We don't want to lose coverage from those tests by completely ignoring them forever.
- We want to have a way to track whether or not a test became stable.
Background and Motivation
Flaky tests are part of every big project that people need to deal with it. Today, there is no good way to manage and track flaky tests, and different efforts and approaches are getting invented by individual teams. None of the approaches that exist today is very ideal as well.
Proposed Feature
The test platform should have the knowledge of a test being known to be flaky. Such tests should be run, but shouldn't fail the test run. For that, such tests will have "inconclusive" result. But there we have two different kind of "inconclusive" results. First is "passed inconclusive", second is "failed inconclusive".
How this is going to work
Individual test frameworks who want to support this feature will:
- Ship their own attribute, let's say it's named FlakyTestAttribute or InconclusiveResultAttribute (that's a choice for the individual test frameworks to make)
- Have a way to let MTP know that the test is marked as such.
- MTP reporting provides a way to get the results of such tests.
- Because of backcompat concerns for TRX, we might need a different way.
- This could be a simple json file that doesn't have the full test reporting capabilities. The json will just have two arrays, one for "inconclusive passed" and the other for "inconclusive failed" and the new json file format won't be concerned about all other kind of tests, and won't be concerned by attachments, display names, duration, traits, ....
- The info needed might be just the TestUid and FQN.
This allows users to track how their flaky tests performs without fully skipping them, and also by keeping them running in the same process as the other tests.
People can then build their system that deals with those json files.
Taking this a step further
There are two additional ways to take this a step further.
-
Manage a flaky info storage on Azure DevOps.
- We could use Azure DevOps to store information about those flaky tests and have an easy way to do queries like "tests that haven't failed in the past X runs" or "tests that haven't failed in the past X days" or "tests with failure rate >= X%" or "tests with failure rate <= X%", ...
-
Specify a specific flakiness reason.
-
This is about having specific failure(s) to be considered "inconclusive failed".
-
Let's say:
[FlakyTest(nameof(IsTimeout))] public void MyTest() { ... } public static void IsTimeout(Exception testFailureException) => testFailureException.Message.Contains("Timed out");
-
In the above scenario, we could have 3 outcomes:
- Test is passing -> Consider as "inconclusive passed".
- Test is failing matching the known flakiness reason (e.g, timeout check in example) -> Consider as "inconclusive failed".
- Test is failing with an unknown reason -> Consider as "failed" and fail the test run.
-
Why is this nice?
- This allows users to still run their flaky tests without failing CI much for "known reasons" that users acknowledged. But still fail in case it's a different failure. That way, the test coverage isn't fully lost, and regressions can still be captured by those tests. So we get the best of everything, CI stability and capturing new potential regressions.
-
The way how the "is known flakiness reason" is implemented will be up to the test frameworks. The test platform should provide some way for the test framework to hook the implementation. The right API shape for this might require some experimentation with implementation.
-
Additional notes
As of today, repos could have their own custom "quarantine" system. The way this works is that tests are by default "unquarantined", and with the existence of a specific test property (via attribute), the test is marked as "quarantined".
Repos could then setup the necessary infrastructure to have two CI pipelines, one that's required and runs only quarantined tests, and another that's not required and runs only unquarantined tests.
This has down sides:
- The detection for whether or not a test is stable cannot be reliable when quarantined tests are isolated from other tests.
- It's extra infrastructure that people have to maintain and potentially duplicate across repos.
Note that the above system is implemented by aspnetcore repo already, but isn't reliable IMO.
Activity
Youssef1313 commentedon Jun 2, 2026
Jakub Jareš (@nohwnd) Up to you if you want to consider implementing this for VSTest as well.
Evangelink commentedon Jun 2, 2026
For context on what is already in the repo today: #8298 (merged) provides a partial, AzDO-only slice of this proposal. It lives entirely in
Microsoft.Testing.Extensions.AzureDevOpsReportand does not introduce any platform- or framework-level concept of a "flaky" test.Already supported (via
Microsoft.Testing.Extensions.AzureDevOpsReport):--report-azdo-flaky-history <N>— queries AzDO REST history for the last N days and decorates failures in the AzDO log with[flaky: failed K/N in last Md]or[REGRESSION](the latter only fires once there are ≥5 prior samples).--report-azdo-quarantine-file <path>— a static text file of FQNs / globs (#comments allowed). Listed tests have their failures demoted towarningin the AzDO log, tagged[quarantined], and emit a##vso[build.addbuildtag]has-quarantined-test-failurebuild tag once per run.--report-azdo-demote-known-flaky— combined with--report-azdo-flaky-history, automatically demotes failures with a ≥25% historical flake rate towarning(default OFF, annotate-only).Retry-After, AOT-safeJsonSerializerContext), 30s bounded history load, all errors swallowed so they can never fail the test run.What this does not do (and what is therefore still in scope for this proposal):
FlakyTestAttribute(or equivalent) on the MSTest side.InconclusivePassed/InconclusiveFailed); the currentTestNodeStatePropertyset is unchanged. A demoted-to-warning failure in AzDO is still a regularFailedTestNodeStatePropertyeverywhere else (terminal, TRX, exit code,--maximum-failed-testsaccounting…).inconclusive-passed/inconclusive-failedarrays ofTestUid+ FQN suggested above).[FlakyTest(nameof(IsTimeout))]and the three-outcome model (passed → inconclusive-passed, known-reason failure → inconclusive-failed, unknown failure → fail the run) are not implemented anywhere.So #8298 mostly addresses the optional "Manage a flaky info storage on Azure DevOps" sub-bullet, but only as AzDO log decoration. The core platform/framework integration described in this proposal (and especially the per-reason predicate) is still entirely unimplemented.
jjonescz commentedon Jun 5, 2026
It might be good to have a way to mark flaky tests without the need to actually change the code (adding FlakyTestAttribute). E.g., make it possible to store the list of flaky tests somewhere else. Perhaps the test platform would not care where, it would just provide an extensibility for this. Then a system like Build Analysis Known Issues could be connected to this.
Also, should there be an option to automatically rerun flaky tests? That might be simpler than needing the whole quarantined/unquarantined pipeline split - just have one pipeline as usual, require all tests to pass, but re-run flaky tests.
Evangelink commentedon Jun 5, 2026
MTP has a retry extension (see https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-retry).
This is still not released but as specified above, I have added some new functionalities to the azdo extension, including:
So we can know which test was retried (correlated with retry extension)
Youssef1313 commentedon Jun 5, 2026
Retry extensions re-tries all failures. I think the request here is to retry tests known to be flaky, and the end user will have an extensibility point to define what tests are flaky. As far as the test platform is concerned, it just needs the info from the test framework if a test is flaky or not. Then it's up to test framework authors to provide the extensibility points for defining flaky tests (this could be attribute as proposed in my original description, or it could be some other arbitrary source which is what Jan proposed, which I think is a great extensibility point to have and allows the flaky test management to be more flexible).
With the proposal above, failing flaky tests don't fail the test run, so retrying could be pointless.
However, there could be a strict mode to say "retry flaky test until it passes", which is very possible today without any changes on the test platform as it's purely a test framework feature. For example, MSTest already has
RetryAttribute(and xUnit could provide one if they don't already). Note that we also provideRetryBaseAttributewhich you could inherit from to define when a test should be retried (so that you only retry when a "known" failure happens - and don't retry on any other failures which could mask real issues)Youssef1313 commentedon Jun 5, 2026
Another feedback I got here is to have easier ways to help with flaky test investigations, so the tests get fixed instead of getting quarantined. I personally see value of both.
Youssef1313 commentedon Jun 5, 2026
One big challenge that we will need to think about is how to handle tests that hang or crash the process.
This is a case where running in the same process and ignoring the test result might not be sufficient. We can decide that these are out of scope for this feature and these tests need to be permanently ignored/skipped.
Anyways, these tests can be problematic today with the alternative quarantine approach that some pipelines do.
Viktor Hofer (@ViktorHofer) William Godbe (@wtgodbe) I wonder how these cases are handled by the recent work done in msbuild and also by the quarantine system in aspnetcore? If the quarantine job hangs or even crashes before executing all the tests. How do you determine whether some tests became stable or not. If those systems rely on parsing TRX, you might get a partial TRX today in these cases. Do you treat a test that didn't report result in TRX as "failing" (which means some tests could have become stable but won't be detected) or do you treat them as "passing" (which means some tests could have been still flaky but will be seen as stable)
jaredpar commentedon Jun 5, 2026
This proposal doesn't seem to address the case where the environment around a test is flaky, not the test itself. For example:
dotnet/runtimeused to run 20+ million tests in CI (not a typo). At that scale even though the product and test code was sound there were almost always a handful of failures that popped up.I definitely agree that the ability to mark specific tests as flaky and have flexibility around them is important. But to truly handle the inherent flakiness of these complex systems we need a way to monitor the entire test suite over a period of time. Ship or no ship decisions need to be able to look at the history of the tests, set metrics for build success (fail build if more than X% of tests fail), and be accessible to agents to track down persistent sources of flakiness.
wtgodbe commentedon Jun 5, 2026
For aspnetcore, we just exclude those results. The threshold is that the test must have ran (and returned results) in at least 66% of all quarantine jobs over the past 30 days, and it must have passed in all of those runs
Youssef1313 commentedon Jun 5, 2026
That's correct in many cases, indeed. I'm trying to think more about concrete ideas that the test platform can do here. I'm not sure if this could end up being a different problem to be solved separately, or if there are good viable ideas for the test platform to squeeze everything under the same system.
Definitely. I wonder if implementing something like a custom
RetryBaseAttribute(MSTest-specific attribute) could be suitable in those cases. The custom retry implementation can detect certain kind of errors and only retry the test for those known transient issues. Note that while this is MSTest-only feature, other frameworks (e.g, xUnit/NUnit) can obviously ship similar features if maintainers wanted do.Integration tests (especially UI testing) is particularly more complicated because I don't think the intermittent failures will have well-known patterns, and I would be afraid that blindly retrying in-process could mask real bugs (e.g, race conditions in product code).
Is the "Manage a flaky info storage on Azure DevOps." part of the proposal not enough for this? Let's put aside that the proposal is suggesting it specifically for AzDO, that could be an arbitrary extension point with some built-in implementations (e.g, AzDO).
Yup, if we have the storage with all the info about what failed, we can easily build some UI to visualize all that and provide the needed metrics for decision making.
Do you mean across history over a period of time here, or in a single run?
If it's over a period of time, that can always be the human decision anyways I think (i.e, ignore test failures and just ship the product). Or do you mean in the context of something like release pipelines where the expected acceptance criteria are defined beforehand, and as long as the history passes those defined criteria, the pipeline goes green and ship without a human reviewing and force-shipping the product?
Youssef1313 commentedon Jun 5, 2026
That means in case of hangs/crashes, tests that didn't get the chance of running, will never get unquarantined even if they got stable, because they never appear in the returned results due to earlier test hanging/crashing.
I have a feeling that flaky tests in the form of hang/crashes should be treated differently and they just have to be skipped completely. I don't see a good solution for those.