[Amaury Levé (Evangelink)]

Summary

Enable affected-test selection in the Windows Release CI lane using Microsoft.Testing.Extensions.AffectedTests 18.12.0-preview.26473.3 and its provider-specific Azure DevOps storage package.

The dependency is version-aligned with Microsoft.Testing.Extensions.CodeCoverage. The repository's hand-authored MTP entry points explicitly register the affected-tests hook for every TFM, including .NET Framework through the package's netstandard2.0 assets.

Storage design

• global.json selects azureDevOpsArtifact storage for public pipeline definition 209.
• Trusted main builds publish versioned per-module shards to the TestFx_AffectedTestsMaps build artifact.
• PR builds query successful definition-209 builds and use the newest commit-valid artifact.
• Pipeline identity and credentials are supplied through CTS_* environment variables and the scoped System.AccessToken; no secret is stored in global.json.
• Missing, stale, incompatible, or inaccessible artifacts conservatively run all tests.

CI behavior

• Trusted main Windows Release runs --collect-test-map.
• Windows Release pull requests run --affected-tests across all repository test TFMs.
• Full-test fallback remains for extension/process failures, scheduled runs, and manual runs.
• Selected-test partial coverage is not published as the repository coverage report.
• eng/validate-affected-tests.ps1 validates package alignment, hook registration, Azure DevOps storage configuration, pipeline identity, all-TFM command wiring, and fallbacks.

Validation

• Release build succeeds with 0 errors using the repo-local toolchain and binary logging.
• Affected-test rollout validation passes.
• 18.12.0-preview.26473.3 package contents include matching netstandard2.0 and net8.0 assets for AffectedTests, CodeCoverage, and the Azure DevOps provider.
• A rebuilt net48 test application contains the new package assets and exposes both --collect-test-map and --affected-tests.
• net8 test applications expose both affected-test commands.
• C# UTF-8 BOM checks, XML/JSON parsing, and git diff --check pass.

[Copilot]

Copilot review overview

🟡 Changes recommended

The fallback loses non-core coverage, and storage validation remains inactive while the feature is disabled.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2

Open (2)

• Preserve non-core coverage in the fallback · New
• Validate storage before the activation switch · New

What changed in this PR

Prepares affected-test selection for MTP-based Windows CI while keeping activation disabled pending the upstream discovery fix.

Changes:

• Adds the affected-tests package and registers it in hand-authored test entry points.
• Adds cached map collection, framework splitting, selection, and fallback pipeline paths.
• Updates rollout validation and documentation.

File	Description
test/​UnitTests/​TestFramework.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​MSTestAdapter.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​MSTestAdapter.PlatformServices.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​MSTest.SourceGeneration.UnitTests/​Program.cs	Registers affected-tests.
test/​UnitTests/​MSTest.SelfRealExamples.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​MSTest.Analyzers.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Platform.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Platform.TerminalReporterContract.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Platform.ServerMode.Client.Sources.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Platform.MSBuild.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Platform.DotnetTestProtocolContract.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Extensions.VSTestBridge.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​UnitTests/​Microsoft.Testing.Extensions.UnitTests/​Program.cs	Registers affected-tests on modern .NET.
test/​IntegrationTests/​PlatformServices.Desktop.IntegrationTests/​Program.cs	Adds framework-guarded registration.
test/​IntegrationTests/​MSTest.Acceptance.IntegrationTests/​Program.cs	Registers affected-tests for acceptance CI.
test/​IntegrationTests/​Microsoft.Testing.Platform.Acceptance.IntegrationTests/​Program.cs	Registers affected-tests for acceptance CI.
samples/​Playground/​Program.cs	Adds framework-guarded registration.
samples/​NUnitPlayground/​Program.cs	Registers affected-tests.
samples/​FSharpPlayground/​Program.fs	Registers affected-tests.
samples/​CtrfPlayground/​Mtp/​Program.cs	Registers affected-tests.
global.json	Configures local mapping storage.
eng/​validate-affected-tests.ps1	Expands rollout wiring validation.
eng/​pipelines/​steps/​test-windows-configuration-tests.yml	Adds framework-split collection, selection, and fallback.
docs/​affected-test-selection.md	Documents the prepared rollout and blocker.
Directory.Packages.props	Centrally versions the new package.
Directory.Build.targets	References the package and filters frameworks.
azure-pipelines.yml	Documents why activation remains disabled.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Package evaluation, multi-target dispatch, and fallback coverage issues must be corrected before activation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Include packages before the application flag is assigned · New
• Preserve non-core coverage in the fallback

Resolved since last review (1)

• Validate storage before the activation switch

Previously missed (1)

In code that hasn't changed since last review

Keep the outer multi-targeting build enabled

Directory.Build.targets:85

For a multi-targeted project, the outer evaluation has no TargetFramework and its identifier is not .NETCoreApp, so AffectedTestsCoreOnly=true sets both application flags to false before MTP can dispatch to the inner builds. The collect, selected-test, and fallback commands consequently omit every multi-targeted project's net8/net9 tests. Apply this filter only to inner evaluations with a concrete target framework.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Test selection now depends on a preview external extension and cross-build artifact semantics that require final human CI verification.

Review effort: Balanced
Findings: 1

Open (1)

• Include packages before the application flag is assigned

Resolved since last review (1)

• Preserve non-core coverage in the fallback

[Copilot]

Copilot review overview

🔵 Needs a closer look

It changes CI test completeness through an experimental extension and external artifact-storage behavior requiring final human validation.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Split synchronization refactor from unrelated changes
• Update stale comment describing affected-test fallback behavior · New

[github-actions]

Final test-triage resolution — Azure Pipelines build 1610624

This is the final resolution for the completed build (analysis: full), superseding any earlier preliminary comment for this build.

Summary

• Affected leg: Windows Release → Test affected changes (the --affected-tests selection step this PR introduces).
• Scope: 186 failed CTRF/JUnit records (same count as the prior build 1610333 analysis), covering the same ~90 distinct test methods/TFM combinations across Microsoft.Testing.Platform.Acceptance.IntegrationTests, MSTest.Acceptance.IntegrationTests, plus 3 records (net462/net8.0/net9.0) for DeadlineStateIsIsolatedBetweenServerRequests in Microsoft.Testing.Platform.UnitTests.
• Signature (unchanged): 174/186 fail with expected: 8, actual: 0 from AssertExitCodeIs / Assert.AreEqual(exitCode, testHostResult.ExitCode), with captured test-host stdout showing "Test run summary: Zero tests ran" (total: 0) even though the acceptance test expects a specific non-zero outcome. The remaining 12 (FailedThreshold_* controller/coverage-threshold tests) show the related expected: 0, actual: 14 propagation pattern, each recorded across 2 retry attempts with no fail-then-pass recovery (both attempts fail identically — not flakiness).
• DeadlineStateIsIsolatedBetweenServerRequests: confirmed pre-existing/unrelated — history.json shows 18/18 prior passing samples across 8 recent main builds, so this is not a regression tied to the affected-tests change; likely incidental noise in this run (single occurrence, no stable new signature isolated in this evidence).

Root cause assessment (confidence: high, unchanged from prior analysis)

This PR's own newly-added builder.AddAffectedTestsProvider() hook in hand-authored MTP Program.cs entry points is inherited by every generated acceptance-test host built at test time. The affected-tests filter (Microsoft.Testing.Extensions.AffectedTests 18.12.0-preview.26473.3, ADO artifact storage) is therefore applied inside child test-host processes the acceptance tests spawn, not just the outer dotnet test invocation, causing over-filtering / zero-test discovery in those nested hosts. The commit history since build 1610333 (Prevent affected-test state from leaking to child hosts, Use affected-tests builder extension, Normalize affected-test registrations) shows active work on exactly this leak, but the current head (039bc09d) still reproduces the identical 186-failure signature, so the fix is not yet effective (or not yet covering the Windows Release acceptance-test lane).

history.json remains incomplete (artifact-download-failed, download-budget-exhausted); combined with PR history being limited to failures/retries, I cannot fully re-confirm recurrence depth beyond the DeadlineState check above, but the 8 sampled main builds show 0 occurrences of the expected: 8, actual: 0 signature, consistent with it being introduced/still present only on this PR's branch.

Disposition

Same as the prior resolution for this PR: this is a one-off failure tied only to the current pull request's own newly introduced CI change, with no durable multi-build recurrence outside this PR. It does not meet the persistent-failure, flaky, crash/hang, or slowness issue thresholds. No issue is warranted. No matching open or closed issue was found for this signature.

Next concrete step: the affected-tests-leak mitigation commits on this branch (6017436d, 9d378afb) have not yet eliminated the "Zero tests ran" pattern in the Windows Release acceptance-test lane — verify the child-host suppression (e.g. stripping DOTNET_CLI_ENABLE_AFFECTED_TESTS/related command-line args, or gating AddAffectedTestsProvider()) actually reaches the generated test assets under Microsoft.Testing.Platform.Acceptance.IntegrationTests before re-running CI.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · 84.2 AIC · ⌖ 4.86 AIC · ⊞ 8.9K

[Copilot]

Copilot review overview

🔵 Needs a closer look

The rollout depends on preview packages and live Azure DevOps artifact behavior that requires final human CI verification.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Split synchronization refactor from unrelated changes
• Update stale comment describing affected-test fallback behavior

[Copilot]

Copilot review overview

🔵 Needs a closer look

Activation depends on preview extension and Azure DevOps artifact behavior that requires final human validation.

Review effort: Balanced
Findings: None

Resolved since last review (2)

• Split synchronization refactor from unrelated changes
• Update stale comment describing affected-test fallback behavior

[github-actions]

🧪 Expert test review — PR #11364

Only one test method's source changed in this PR: ServerTests.DeadlineStateIsIsolatedBetweenServerRequests (test/UnitTests/Microsoft.Testing.Platform.UnitTests/ServerMode/ServerTests.cs). All other diffs in the PR touch Program.cs entry points, pipeline/MSBuild configuration, docs, and a WellKnownEnvironmentVariables allow-list — none of which are test methods.

Grade	Test	Mutation	Notes	How to improve
A (90–100)	mod ServerTests. DeadlineStateIsIsolatedBetweenServerRequests	2/2 killed	Correctly re-targets assertions to GetProcessExitCodeWithoutIgnore() so the new ignore-exit-code policy (added elsewhere in this PR) doesn't mask the deadline-isolation verdict being tested; both isolated-state equality assertions still fail on the relevant mutations.	—

The diff is a small, surgical adaptation: it casts the exit-code service to the concrete TestApplicationResult so the test can call the internal GetProcessExitCodeWithoutIgnore() instead of the public GetProcessExitCode(), which now applies the new --ignore-exit-code/TESTINGPLATFORM_EXITCODE_IGNORE policy from ExitCodeIgnorePolicy. Since the test asserts per-request deadline-state isolation (not the ignore-policy behavior), bypassing the ignore filter keeps the assertions meaningful and correctly scoped. No anti-patterns, assertion weakening, or structural regressions were introduced by this change.

_{This advisory comment was generated automatically. Grades are heuristic and informational — they do not block merging. Suggestions on the Files changed tab can be applied with one click. Re-run with /review-tests.}

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

🤖 Automated content by GitHub Copilot. Generated by the Test Reviewer on PR (on open / sync) workflow. · copilot · auto · 135.1 AIC · ⌖ 9.74 AIC · ⊞ 17.9K · ◷