Fix FileSystemWatcher test flakiness on Windows and re-enable 86 disabled tests

[danmoseley]

Categories of failures

The FSW test failures on Windows fall into three categories:

1. Timeout too short for callback latency — Under thread pool pressure (typical in CI), ReadDirectoryChangesW callbacks arrive late. The 1000ms WaitForExpectedEventTimeout expires before the event is delivered. This is the dominant failure mode: "<EventType> event did not occur as expected".

2. Stale state between retries — Some tests create files/directories in action() but never clean up. When ExpectEvent retries (re-creating the watcher and re-running the action), the second attempt fails because the file already exists, or the watcher fires on leftover state rather than the fresh action.

3. ExpectNoEvent using wrong timeout — ExpectNoEvent was hardcoded to use WaitForExpectedEventTimeout (1000ms) instead of WaitForUnexpectedEventTimeout (500ms), making negative tests take 2x longer than intended and potentially masking timing issues.

Fixes in this PR (and why each)

No product changes. All changes are in the test utility (FileSystemWatcherTest.cs) and individual test files.

1. Progressive retry timeouts — ExpectEvent now multiplies WaitForExpectedEventTimeout by the attempt number (1x, 2x, 3x). This directly addresses category 1: if the first attempt's 1000ms is too short under load, the second attempt waits 2000ms and the third waits 3000ms, giving the callback time to arrive without penalizing the common fast case.

2. Increased SubsequentExpectedWait (10ms to 500ms) — Tests that check multiple event types (e.g., Changed + Created) waited only 10ms for the second event type after catching the first. Under load, the second event arrives later. 500ms provides adequate margin.

3. Settling delay after enabling watcher (50ms Thread.Sleep) — Added in ExecuteAndVerifyEvents, ExpectEvents, and TryErrorEvent after setting EnableRaisingEvents = true and before running the test action. While Windows registers the watch synchronously, this guards against edge cases on all platforms.

4. Cleanup in Create event tests — File.Create.cs and Directory.Create.cs tests now delete the created file/directory in a finally block, so retries start from a clean state (category 2).

5. ExpectNoEvent timeout fix — Changed to use WaitForUnexpectedEventTimeout as intended (category 3).

6. Re-enabled 86 tests previously disabled by 25 [ActiveIssue] annotations (11 class-level, 14 method-level) referencing Failing test due to no detected IO events in 'System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents' #103584 on Windows. Two tests (File_Move_With_Set_Environment_Variable, File_Move_With_Unset_Environment_Variable) are disabled against a new issue for a product-level bug unrelated to timing.

Validation steps

Tested on Intel i9-14900K (24 cores / 32 threads), 64 GB RAM, Windows 11. Tests run on both NTFS and ReFS (dev drive).

Controlled callback delay (proving the fix addresses the root cause):

• Injected a temporary busy-spin delay into ReadDirectoryChangesCallback to deterministically reproduce the late-event-delivery condition that occurs under thread pool starvation in CI
• At 900ms delay, baseline (no fixes): 114 event failures across 20 runs (100% failure rate)
• At 900ms delay, with fixes: 0 event failures across 20 runs

Stress testing:

• Concurrent stress generator running alongside tests: 32 CPU-saturating workers (one per logical processor) doing tight math loops, 8 IO workers (rapid file create/write/delete cycles with 1-64KB random data), and continuous thread pool flooding (~500 short work items queued every 10ms) to simulate thread pool contention
• No matter how much stress, could not reproduce failures even without fixes - this local machine is just too fast compared to a CI machine.
• Under this load, ran 20 sequential iterations on each filesystem with fixes: 0 flaky failures on NTFS, 0 on ReFS
• 3 parallel test instances running simultaneously under stress with fixes: 0 flaky failures

Unchanged

• Linux/macOS — Separate issues exist (FileSystemWatcher SymbolicLink tests failing after inotify refactor (#117148) #125737 Linux inotify race after FileSystemWatcher.Linux: use a single inotify instance and refactor watch tracking. #117148 refactor; System.IO.Tests.File_Move_Tests.File_Move_Multiple_From_Watched_To_Unwatched_Mac test failure #125018 macOS duplicate FSEvents). These have different root causes and are not addressed here.
• Product code — No changes to FileSystemWatcher implementation. The callback delivery latency is fundamentally an OS/thread-pool scheduling issue, not a product bug. The fix is making tests tolerant of realistic delivery variance.
• Issue FileSystemWatcher_DirectorySymbolicLink_TargetsFile_Fails #53366 (SymbolicLink / InternalBufferSize tests) — Remains disabled on all platforms; this is a separate product-level issue.

Fixes #103630
Fixes #103584

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR targets Windows flakiness in System.IO.FileSystem.Watcher tests by increasing tolerance to callback latency and retry-related stale state, and then re-enables a large set of tests previously disabled on Windows.

Changes:

• Adjusts FileSystemWatcherTest timeouts/retry behavior (including progressive timeouts) and fixes ExpectNoEvent default timeout usage.
• Adds a small post-EnableRaisingEvents settling delay to reduce timing sensitivity across platforms.
• Re-enables many tests previously disabled via [ActiveIssue], and adds cleanup in some create-related tests to keep retries idempotent.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs	Updates core test helper timeouts/retry logic and adds settling delays.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs	Removes Windows [ActiveIssue] suppressions and adds cleanup for create-related retry safety.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.cs	Re-enables a previously disabled init/resume test on Windows.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.SymbolicLink.cs	Re-maps Windows ActiveIssue annotations for specific symlink tests.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.InternalBufferSize.cs	Removes Windows [ActiveIssue] suppression for internal buffer size tests.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.NotifyFilter.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Delete.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Create.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Changed.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.NotifyFilter.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Move.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Delete.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Create.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Changed.cs	Removes Windows [ActiveIssue] suppression at the class level.

Comments suppressed due to low confidence (1)

src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:286

• cleanup() is executed after the assertion. If the assertion fails (i.e., an unexpected event did occur), cleanup won’t run, which can leave state behind for the outer retry wrapper and lead to follow-on failures. Run cleanup in a finally so it executes regardless of assertion outcome.

            bool result = ExecuteAndVerifyEvents(watcher, unExpectedEvents, action, false, expectedPath == null ? null : new string[] { expectedPath }, timeout);
            Assert.False(result, "Expected Event occurred");

            if (cleanup != null)
                cleanup();

[stephentoub]

If 1s is flaky, why do we expect 2s to never be flaky?

[danmoseley]

Could we not say the same thing for every timeout and sleep in all our tests? It's hard to imagine one that couldn't be defeated by unlucky thread scheduling?

(Early in my career, I was told "every sleep in a test is a test bug" but that was testing UI, where we were to instead wait indefinitely for objects/events)

[stephentoub]

Could we not say the same thing for every timeout and sleep in all our tests?

Certainly ones that fail a test after 2s. We have lots of failsafe timeouts of, say, 60s where if the operation under test runs that long, that's unacceptable under any circumstance and the test fails. But 2s is easy to accidentally trigger in CI, especially if we already have evidence that we're hitting 1s.

[danmoseley]

Gotcha, OK I'll see whether there's scope to raise these more, if it's sensible.

[stephentoub]

If this is necessary, it's also going to be flaky.

[danmoseley]

Couldn't you say the same thing about most retries?

[danmoseley]

I think fundamentally, we know that testing FSW (on any OS apparently) is not something we know how to do deterministically. There are some correctness fixes in here (actually delete the file before retrying) but to get the tests to be much more robust, it's just sleeps and retries.

So we have two options

• increase sleep and retry strategies until they are acceptably reliable, or
• leave tests disabled in most cases, move into some requested outer loop or even just only run them if you touch relevant product code.

LMK if you have another suggestion.

For some reason I have a bias in my head that retries are more robust than sleeps, but ultimately they both involve picking an arbitrary number we hope is large enough.