Fix flaky DistributedFileLoggerParameters test with test isolation#13669
Fix flaky DistributedFileLoggerParameters test with test isolation#13669jankratochvilcz wants to merge 2 commits intomainfrom
Conversation
Both DistributedFileLoggerParameters and DistributedLoggerNullEmpty wrote msbuild0.log to the shared working directory, causing file contention when tests ran concurrently. The old finally block also never cleaned up msbuild0.log. Use TestEnvironment with an isolated temp folder and SetCurrentDirectory so each test run operates on its own files. TestEnvironment.Dispose handles all cleanup automatically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Addresses CI flakiness in FileLogger_Tests caused by multiple distributed-logger tests contending on the same log file in a shared working directory.
Changes:
- Added
ITestOutputHelperinjection to the test class for captured diagnostic output. - Updated distributed-logger tests to use
TestEnvironment+ an isolated temp folder, and set the current directory per test to avoid log-file collisions. - Removed brittle manual cleanup logic in favor of
TestEnvironment-managed cleanup.
| } | ||
| fileLogger.NodeId = 0; | ||
| fileLogger.Initialize(new EventSourceSink()); | ||
| Assert.Equal(0, string.Compare(fileLogger.InternalFilelogger.Parameters, "ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;", StringComparison.OrdinalIgnoreCase)); |
There was a problem hiding this comment.
In this modified test code, assertions are still using xUnit Assert.*. This repo's Build.UnitTests generally uses Shouldly for assertions; please convert the updated assertions in this method (e.g., around this line and the similar comparisons below) to Shouldly and add the corresponding using Shouldly; if needed.
There was a problem hiding this comment.
Review Summary
Overall: Good fix ✅ — the flakiness root cause is correctly diagnosed and addressed.
What's Good
- ✅
TestEnvironment+TransientTestFolder+SetCurrentDirectoryis the idiomatic MSBuild test isolation pattern - ✅
usingdeclaration ensures deterministic cleanup even if the test throws - ✅ Removes the brittle manual
try/finallycleanup that missedmsbuild0.log - ✅ Both
DistributedFileLoggerParametersandDistributedLoggerNullEmptyare isolated - ✅
ITestOutputHelperproperly injected for diagnostic output in CI
Suggestion (🟡 Important)
- The assertions use the
Assert.Equal(0, string.Compare(...))pattern which should be replaced with Shouldly (ShouldBe(..., StringCompareShould.IgnoreCase)) per repo testing conventions for modified code. This also provides better failure diagnostics — the current pattern only reports "Expected 0, got 1" on failure, hiding what the actual string difference was.
Generated by Expert Code Review (on open) for issue #13669 · ● 4.6M
| fileLogger.NodeId = 0; | ||
| fileLogger.Initialize(new EventSourceSink()); | ||
| Assert.Equal(0, string.Compare(fileLogger.InternalFilelogger.Parameters, "ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;", StringComparison.OrdinalIgnoreCase)); |
There was a problem hiding this comment.
🟡 Convention (Important): Per repo testing conventions, modified code should use Shouldly assertions even when the rest of the file uses xUnit. The Assert.Equal(0, string.Compare(..., StringComparison.OrdinalIgnoreCase)) pattern is verbose and produces poor failure messages. Consider replacing with:
fileLogger.InternalFilelogger.Parameters.ShouldBe(
"ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;",
StringCompareShould.IgnoreCase);This applies to all four assertions in this method. You'd need to add using Shouldly; at the top of the file.
The Shouldly version is more readable and on failure will show the actual vs expected value, rather than just "Expected: 0, Actual: 1" which makes debugging much harder.
| fileLogger.Shutdown(); | ||
|
|
||
| fileLogger.NodeId = 4; | ||
| fileLogger.Parameters = "logfile=" + Path.Combine(folder.Path, "mylogfile.log"); |
There was a problem hiding this comment.
🟡 Assertion style: replace Assert.Equal(0, string.Compare(...)) with Shouldly
Per repo testing conventions, Shouldly assertions should be used in modified code. The current pattern is verbose and produces poor failure diagnostics. Replace all occurrences in this method with:
fileLogger.InternalFilelogger.Parameters.ShouldBe(
"ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;",
StringCompareShould.IgnoreCase);This applies to all 4 assertion blocks in DistributedFileLoggerParameters.
Replace Assert.Equal(0, string.Compare(..., StringComparison.OrdinalIgnoreCase)) with str.ShouldBe(expected, StringCompareShould.IgnoreCase) per repo testing conventions. Shouldly produces clearer failure diagnostics than the xUnit + string.Compare pattern. The reviewer's note about a missing 'using Xunit.Abstractions' does not apply: the project uses xUnit v3, where ITestOutputHelper lives in the Xunit namespace and is already imported. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Addressed the review feedback in a9e6760:
Verified locally: @copilot please re-review. |
Re-reviewed and verified. Commit a9e6760 already addresses the feedback correctly ( |
Problem
DistributedFileLoggerParametersintermittently fails in CI with:Both
DistributedFileLoggerParametersandDistributedLoggerNullEmptycreatemsbuild0.login the shared working directory. When tests run concurrently they contend on the same file. The oldfinallyblock also never cleaned upmsbuild0.logitself.Fix
TestEnvironmentwith an isolated temp folder andSetCurrentDirectoryso each test operates on its own filesTestEnvironment.Dispose()handles all cleanup automatically, replacing the brittle manualtry/finallyITestOutputHelperto the test class for proper test output capture