Skip to content

Fix flaky DistributedFileLoggerParameters test with test isolation#13669

Open
jankratochvilcz wants to merge 2 commits intomainfrom
fix/flaky-distributed-file-logger-test
Open

Fix flaky DistributedFileLoggerParameters test with test isolation#13669
jankratochvilcz wants to merge 2 commits intomainfrom
fix/flaky-distributed-file-logger-test

Conversation

@jankratochvilcz
Copy link
Copy Markdown
Contributor

Problem

DistributedFileLoggerParameters intermittently fails in CI with:

LoggerException: Failed to write to log file "msbuild0.log". The process cannot access the file because it is being used by another process.

Both DistributedFileLoggerParameters and DistributedLoggerNullEmpty create msbuild0.log in the shared working directory. When tests run concurrently they contend on the same file. The old finally block also never cleaned up msbuild0.log itself.

Fix

  • Use TestEnvironment with an isolated temp folder and SetCurrentDirectory so each test operates on its own files
  • TestEnvironment.Dispose() handles all cleanup automatically, replacing the brittle manual try/finally
  • Added ITestOutputHelper to the test class for proper test output capture

Both DistributedFileLoggerParameters and DistributedLoggerNullEmpty wrote
msbuild0.log to the shared working directory, causing file contention when
tests ran concurrently. The old finally block also never cleaned up
msbuild0.log.

Use TestEnvironment with an isolated temp folder and SetCurrentDirectory
so each test run operates on its own files. TestEnvironment.Dispose
handles all cleanup automatically.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 30, 2026 17:15
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Addresses CI flakiness in FileLogger_Tests caused by multiple distributed-logger tests contending on the same log file in a shared working directory.

Changes:

  • Added ITestOutputHelper injection to the test class for captured diagnostic output.
  • Updated distributed-logger tests to use TestEnvironment + an isolated temp folder, and set the current directory per test to avoid log-file collisions.
  • Removed brittle manual cleanup logic in favor of TestEnvironment-managed cleanup.

Comment thread src/Build.UnitTests/FileLogger_Tests.cs Outdated
}
fileLogger.NodeId = 0;
fileLogger.Initialize(new EventSourceSink());
Assert.Equal(0, string.Compare(fileLogger.InternalFilelogger.Parameters, "ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;", StringComparison.OrdinalIgnoreCase));
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this modified test code, assertions are still using xUnit Assert.*. This repo's Build.UnitTests generally uses Shouldly for assertions; please convert the updated assertions in this method (e.g., around this line and the similar comparisons below) to Shouldly and add the corresponding using Shouldly; if needed.

Copilot uses AI. Check for mistakes.
Comment thread src/Build.UnitTests/FileLogger_Tests.cs
Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Overall: Good fix ✅ — the flakiness root cause is correctly diagnosed and addressed.

What's Good

  • TestEnvironment + TransientTestFolder + SetCurrentDirectory is the idiomatic MSBuild test isolation pattern
  • using declaration ensures deterministic cleanup even if the test throws
  • ✅ Removes the brittle manual try/finally cleanup that missed msbuild0.log
  • ✅ Both DistributedFileLoggerParameters and DistributedLoggerNullEmpty are isolated
  • ITestOutputHelper properly injected for diagnostic output in CI

Suggestion (🟡 Important)

  • The assertions use the Assert.Equal(0, string.Compare(...)) pattern which should be replaced with Shouldly (ShouldBe(..., StringCompareShould.IgnoreCase)) per repo testing conventions for modified code. This also provides better failure diagnostics — the current pattern only reports "Expected 0, got 1" on failure, hiding what the actual string difference was.

Generated by Expert Code Review (on open) for issue #13669 · ● 4.6M

Comment thread src/Build.UnitTests/FileLogger_Tests.cs Outdated
Comment on lines +498 to +500
fileLogger.NodeId = 0;
fileLogger.Initialize(new EventSourceSink());
Assert.Equal(0, string.Compare(fileLogger.InternalFilelogger.Parameters, "ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;", StringComparison.OrdinalIgnoreCase));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Convention (Important): Per repo testing conventions, modified code should use Shouldly assertions even when the rest of the file uses xUnit. The Assert.Equal(0, string.Compare(..., StringComparison.OrdinalIgnoreCase)) pattern is verbose and produces poor failure messages. Consider replacing with:

fileLogger.InternalFilelogger.Parameters.ShouldBe(
    "ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;",
    StringCompareShould.IgnoreCase);

This applies to all four assertions in this method. You'd need to add using Shouldly; at the top of the file.

The Shouldly version is more readable and on failure will show the actual vs expected value, rather than just "Expected: 0, Actual: 1" which makes debugging much harder.

fileLogger.Shutdown();

fileLogger.NodeId = 4;
fileLogger.Parameters = "logfile=" + Path.Combine(folder.Path, "mylogfile.log");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Assertion style: replace Assert.Equal(0, string.Compare(...)) with Shouldly

Per repo testing conventions, Shouldly assertions should be used in modified code. The current pattern is verbose and produces poor failure diagnostics. Replace all occurrences in this method with:

fileLogger.InternalFilelogger.Parameters.ShouldBe(
    "ForceNoAlign;ShowEventId;ShowCommandLine;logfile=msbuild0.log;",
    StringCompareShould.IgnoreCase);

This applies to all 4 assertion blocks in DistributedFileLoggerParameters.

Replace Assert.Equal(0, string.Compare(..., StringComparison.OrdinalIgnoreCase))
with str.ShouldBe(expected, StringCompareShould.IgnoreCase) per repo
testing conventions. Shouldly produces clearer failure diagnostics
than the xUnit + string.Compare pattern.

The reviewer's note about a missing 'using Xunit.Abstractions' does
not apply: the project uses xUnit v3, where ITestOutputHelper lives
in the Xunit namespace and is already imported.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jankratochvilcz
Copy link
Copy Markdown
Contributor Author

Addressed the review feedback in a9e6760:

  • ✅ Converted all four Assert.Equal(0, string.Compare(..., StringComparison.OrdinalIgnoreCase)) blocks to str.ShouldBe(expected, StringCompareShould.IgnoreCase) per repo testing conventions.
  • ✅ Added using Shouldly;.
  • 🔵 The note about a missing using Xunit.Abstractions; does not apply — this project is on xUnit v3, where ITestOutputHelper lives in the Xunit namespace (already imported). The file built and the tests passed locally.

Verified locally: DistributedFileLoggerParameters and DistributedLoggerNullEmpty both pass.

@copilot please re-review.

Copy link
Copy Markdown
Contributor

Copilot AI commented May 7, 2026

Addressed the review feedback in a9e6760:

  • ✅ Converted all four Assert.Equal(0, string.Compare(..., StringComparison.OrdinalIgnoreCase)) blocks to `str.ShouldBe(expected, StringCompareShould.Ign...

Re-reviewed and verified. Commit a9e6760 already addresses the feedback correctly (Shouldly assertions are in place, ITestOutputHelper resolves via Xunit in this test project), and both affected tests pass locally with targeted execution: DistributedFileLoggerParameters and DistributedLoggerNullEmpty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants