Skip to content

Move independent MSTest APIs into the core framework - #12412

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
dev/amauryleve/common-framework-api-move
Nov 12, 2026
Merged

Amaury Levé (Evangelink) merged 2 commits into
mainfrom
dev/amauryleve/common-framework-api-move

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Pull request

Summary

Relates to #7225.

Move the dependency-closed common definitions from TestFramework.Extensions into the core framework as a small prerequisite, not the UWP/WinUI package split:

  • MessageLevel.
  • TestRun, ITestRunInfo and PlannedTest.
  • All five PrivateObject partials, PrivateType and RuntimeTypeHelper.
  • The five .NET Framework configuration-helper files.

Scope: 16 source moves, 32 files overall. Thirteen moved files are byte-identical; three change only XML-doc references to the still-separate TestContext. Supporting edits are API tracking, core's .NET Framework System.Configuration reference, one assembly-qualified configuration fixture, one ownership test, and documentation.

Deferred: TestContext, DeploymentItemAttribute, and the test-class factory/lease interfaces remain in Extensions because their move requires UI build/package changes. Existing core TFMs, package ownership/layout/build assets, UI source, adapter/analyzer policy and SDK selection are unchanged.

Breaking change for MSTest 5: moved types change assembly identity and inherit core's existing assembly-level metadata defaults. Extensions specifies ComVisible(false) on non-WinUI builds; core leaves that attribute unset. No core-wide COM policy or compatibility forwarders are introduced. Recompile consumers and update assembly-qualified configuration names; this is not binary-compatible.

Design and decision record

The staging rationale is #7225 and this bounded relocation; no separate feature design record is needed. The existing run-state RFC now reflects its owner and the breaking identity change.

Moving the coupled types now would require UI-flavored core builds and package selection changes. Keeping those out preserves reviewability and the existing consumer composition. Existing shipped API history is retained, with explicit unshipped removals in Extensions and additions in core. Method bodies, target guards and lifecycle behavior are not rewritten.

Demonstration

This is a nonvisual assembly move, so a package-consumer transcript is more useful than screenshots. Real package consumers reported:

CONTRACTS_PASSED|net10.0|MSTest.TestFramework.Extensions
CONTRACTS_PASSED|net10.0|MSTest.TestFramework

The first line uses the saved baseline package; the second uses the head package. TestContext remains in Extensions in both. The package still has exactly 105 entries, and its nuspec is identical except for the source commit, including the .NET Framework System.Configuration declaration.

Validation and regression proof

Baseline: 50b513022be7c356cb342866a3b755a4b18bf97d. Head: a8c7f9560bb43666655d29fe8dfc4c74a7a6798b. Windows, Release, repository SDK 12.0.100-alpha.1.26556.115; .NET Framework assets target net462 and run on the installed .NET Framework runtime.

Check Result
Core and retained Extensions builds Exit 0, no warnings/errors; core netstandard2.0/net462/net10/net11 and all seven existing Extensions targets, including classic/modern UWP and WinUI.
Common contracts: net48 / net10 / net11 / WinUI Before: 34 / 10 / 10 / 8 passed. After: 35 / 11 / 11 / 9 passed. The four additional executions are the new ownership test. No failures or skips.
Adapter TestRunInfoTests 7 passed on each of net462, net10, net11 and retained WinUI: 28 passed, exit 0, no skips.
Packed consumers, both production states Four real processes per state passed on net462, net10, net11 and UseWinUI/net10-windows; netstandard2.0 also compiled.
Package provenance Separate restore caches; restored nupkg and every executed core/Extensions DLL matched the corresponding package SHA-256. Head nuspec identifies the exact head above.

The packed consumers exercised snapshot collection copying, duplicate properties, cross-assembly MessageLevel/TestContext dispatch, cancellation, private instance/static reflection, and old/new assembly-qualified configuration loading. The retained WinUI consumer also checked UI attribute presence and the continued absence of deployment/directory APIs. These are managed binding/consumption checks, not native UI activation.

Executed build/pack commands and test selectors

The normal repository build bootstrap was used, with Windows PowerShell and Git on the native shell PATH:

.\build.cmd -c Release -v quiet -projects test\UnitTests\TestFramework.UnitTests\TestFramework.UnitTests.csproj /bl:artifacts\log\Release\common-move-published-head-build-01.binlog
.\build.cmd -c Release -v quiet -projects test\UnitTests\MSTestAdapter.PlatformServices.UnitTests\MSTestAdapter.PlatformServices.UnitTests.csproj /bl:artifacts\log\Release\common-move-head-adapter-build-01.binlog

Both passed, exit 0. The first was also run before the move.

The owning package was built/packed through the same repository-pinned desktop MSBuild helper, preserving the classic UWP target:

$configuration = 'Release'
$restore = $false
. .\eng\common\tools.ps1
MSBuild (Resolve-Path src\TestFramework\TestFramework.Extensions\TestFramework.Extensions.csproj).Path /t:Build /p:Configuration=Release '/bl:artifacts\log\Release\common-move-published-head-all-targets-{}.binlog'
MSBuild (Resolve-Path src\TestFramework\TestFramework.Extensions\TestFramework.Extensions.csproj).Path /t:Pack /p:Configuration=Release /p:NoBuild=true '/bl:artifacts\log\Release\common-move-published-head-pack-{}.binlog'

Both passed, exit 0. The baseline was packed with the same helper. The earlier command .\build.cmd -c Release -pack -projects src\TestFramework\TestFramework.Extensions\TestFramework.Extensions.csproj /bl:artifacts\log\Release\common-move-baseline-pack-01.binlog produced the baseline package but exited 1 because its product-wide post-pack verifier rejected the seven other MSTest packages not requested by that partial build. No verifier or package machinery was changed.

Direct test-host command shape, with the exact selections used:

& .\.dotnet\dotnet.exe artifacts\bin\TestFramework.UnitTests\Release\net10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.dll --treenode-filter '/*/Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.Extensions/*/*' --report-trx --report-trx-filename final-net10.trx --results-directory artifacts\TestResults\Release\common-move-final-net10 --ansi off --show-test-results failed
& .\.dotnet\dotnet.exe artifacts\bin\TestFramework.UnitTests\Release\net11.0\Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.dll --treenode-filter '/*/Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.Extensions/*/*' --report-trx --report-trx-filename final-net11.trx --results-directory artifacts\TestResults\Release\common-move-final-net11 --ansi off --show-test-results failed
& artifacts\bin\TestFramework.UnitTests\Release\net48\Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.exe --treenode-filter '/*/Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests.Extensions/*/*' --report-trx --report-trx-filename final-net48.trx --results-directory artifacts\TestResults\Release\common-move-final-net48 --ansi off --show-test-results failed

Results: 11, 11 and 35 passed respectively, all exit 0. On net10.0-windows10.0.18362.0\win-x64, the same host used the class selectors ExtensionsContractTests/* and TestContextTests/* within the same namespace: 5 and 4 passed, exit 0. Adapter hosts used /*/Microsoft.VisualStudio.TestPlatform.MSTestAdapter.UnitTests.Execution/TestRunInfoTests/*: 7 passed per target, exit 0.

All nine final TRX reports were parsed: 94 executed, 94 passed, zero failed/skipped. Binlogs, TRX, the paired-run manifest and isolated consumer sources/assets/provenance are retained locally; no hosted artifact was uploaded.

MSTest.TestFramework.5.0.0-dev.nupkg SHA-256:

baseline CE8BA2C0698C104129EBCBE7B48732F44B21D592E68122A99453052C30FB1D89
head     9DF0A9BCEE4042F8CBCE2F7DA769DCA5FA8D1EE60299D2F1C4332D9BC8162994

Limitations: the full acceptance suite and real UWP/WinUI application activation were not run for this bounded move. The broader baseline WinUI extension selection executed 14 tests and had three pre-existing native activation/startup failures (DispatcherLookupPreservesExplicitPrecedenceAndCachedFailure, NativeApplicationStartupInIsolatedProcesses, RealDispatcherCapturesDirectAndEnqueuedResultsAndFailures). Those were not repaired or counted as passing proof. No CI result is claimed.

Test-value evidence

Contract / test case Counterfactual Without the move With the move Evidence / gap
ExtensionsContractTests.CommonDefinitionsAreOwnedByCoreFramework Original production at the baseline SHA, new test retained Executed: 1 failed, exit 2, because MessageLevel belonged to Extensions rather than core Executed: passed on all four test TFMs Test-only comparison patch SHA-256 7693B1DC81A50FB7781B7BFD74E10180B709276D4890933B7396F9CE1E50E38D. This is an ownership assertion failure, not a build/setup failure.
Existing ExtensionsContractTests, TestContextTests, PrivateObjectTests, PrivateTypeTests, RuntimeTypeHelperTests, ConfigurationSettingsTests Original production Executed: 62 preservation cases passed Executed: the same 62 cases passed, plus 4 ownership cases TRX identities and counts reconciled before/after; no weakened assertions.
Packed library consumption and retained WinUI shape Saved baseline package in its own cache Executed: 4 consumers passed; netstandard2.0 compiled Executed: 4 consumers passed; netstandard2.0 compiled Loaded-binary hashes matched each package; native UI activation remains unverified.

Move message-level, run-state, private reflection and configuration definitions without changing core targets or package layout. Keep TestContext, DeploymentItem and factory/lease interfaces in Extensions for the later UI-dependent move.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 21:38
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 11, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12412

Compared merge base 50b513022be7c356cb342866a3b755a4b18bf97d with head a8c7f9560bb43666655d29fe8dfc4c74a7a6798b. Evidence applies to this head, not every intermediate commit.

Test-value evidence

All entries are static predictions. No paired execution records were available, and none were run.

Contract Test / decisive assertion Counterfactual Result
MessageLevel, ITestRunInfo, PlannedTest and TestRun are owned by the core MSTest.TestFramework assembly ExtensionsContractTests.CommonDefinitionsAreOwnedByCoreFramework: type.Assembly.Should().BeSameAs(typeof(TestMethodAttribute).Assembly) If a type were left in TestFramework.Extensions, the assembly identity would differ and the assertion would fail. Static prediction: likely detected
The .NET Framework-only types (PrivateObject, PrivateType, DataSourceElement*, TestConfiguration*) are owned by the core assembly Same test, #if NETFRAMEWORK branch Same as above. The branch only runs on the net462 TFM. Static prediction: likely detected, but only if the net462 unit-test run executes it
The app.config section type resolves from MSTest.TestFramework DataSourceTests (acceptance): the config string now names MSTest.TestFramework If the type stayed in the Extensions assembly, loading the section would fail and the data-source tests would fail. Static prediction. This is an acceptance test and was not executed.
Public/internal API files, PACKAGE.md, the RFC and the specification Documentation and API-tracking files — Not applicable. These are not behavioral.

Evidence gaps:

  • No executed red/green records were accessible.
  • The net462 AdditionalFiles conditions and the System.Configuration reference in TestFramework.csproj are covered only by compilation, not by a behavioral test.

🧵 Parallel safety

STATUS: PARTIAL. The read-only specialist task was not run in this session.

The changed test code gives no sign of a new hazard:

  • CommonDefinitionsAreOwnedByCoreFramework only reads reflection metadata.
  • The DataSourceTests change is a literal string in a generated asset.
  • No parallelization configuration changed (config_count is 0).

This is a manual impression, not a specialist conclusion, and it should not be read as a clean parallel-safety result.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass A (90–100) new ExtensionsContractTests.CommonDefinitionsAreOwnedByCoreFramework An assembly-identity assertion per type covers the ownership contract, with a per-type message. None
Pass B (80–89) mod DataSourceTests (config section type string) This is a data-only change to an asset; the existing acceptance assertions exercise the section load. None

No verified, actionable inline findings were identified, so no inline comments were posted.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~30 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 11, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the relocation of independent MSTest APIs into the core framework.

No open issues 🎉

What changed in this PR

Moves independent MSTest APIs from Extensions into the core framework as a prerequisite to #7225’s UWP/WinUI package split. This intentionally changes assembly identity for MSTest 5; consumers must recompile and update assembly-qualified configuration names. UI packaging and TestContext ownership remain unchanged.

Changes:

  • Relocates 16 source files covering message severity, run-state APIs, private reflection, and .NET Framework configuration helpers.
  • Updates API tracking and adds core’s .NET Framework System.Configuration reference.
  • Adds an assembly-ownership test and updates the configuration fixture and migration documentation.

Test-value evidence: For head a8c7f9560bb43666655d29fe8dfc4c74a7a6798b against baseline 50b513022be7c356cb342866a3b755a4b18bf97d, the PR reports ownership-test red/green results and preservation/package-consumer passes. Execution artifacts were retained locally and were not independently verified here; full acceptance and native UI activation remain unverified.

File Description
test/​UnitTests/​TestFramework.UnitTests/​Extensions/​ExtensionsContractTests.cs Adds core assembly-ownership assertions.
test/​IntegrationTests/​MSTest.Acceptance.IntegrationTests/​DataSourceTests.cs Updates the configuration section’s assembly name.
src/​TestFramework/​TestFramework/​TestRun.cs Moves ambient run-state access into core.
src/​TestFramework/​TestFramework/​TestFramework.csproj Includes net462 API baselines and configuration reference.
src/​TestFramework/​TestFramework/​RuntimeTypeHelper.cs Moves reflection method-selection helpers into core.
src/​TestFramework/​TestFramework/​PublicAPI/​PublicAPI.Unshipped.txt Records common public API additions.
src/​TestFramework/​TestFramework/​PublicAPI/​net462/​PublicAPI.Unshipped.txt Records .NET Framework public API additions.
src/​TestFramework/​TestFramework/​PublicAPI/​net462/​PublicAPI.Shipped.txt Initializes the net462 public API baseline.
src/​TestFramework/​TestFramework/​PrivateType.cs Moves static private-member reflection into core.
src/​TestFramework/​TestFramework/​PrivateObject.Properties.cs Moves private-property access into core.
src/​TestFramework/​TestFramework/​PrivateObject.Invoke.cs Moves private-method invocation into core.
src/​TestFramework/​TestFramework/​PrivateObject.Helpers.cs Moves private-reflection lookup helpers into core.
src/​TestFramework/​TestFramework/​PrivateObject.Fields.cs Moves private-field access into core.
src/​TestFramework/​TestFramework/​PrivateObject.cs Moves the private-object wrapper into core.
src/​TestFramework/​TestFramework/​PlannedTest.cs Relocates planned-test metadata; adjusts XML documentation.
src/​TestFramework/​TestFramework/​MessageLevel.cs Relocates message severity; adjusts XML documentation.
src/​TestFramework/​TestFramework/​ITestRunInfo.cs Relocates the run-information contract; adjusts XML documentation.
src/​TestFramework/​TestFramework/​InternalAPI/​net462/​InternalAPI.Unshipped.txt Records .NET Framework internal API additions.
src/​TestFramework/​TestFramework/​InternalAPI/​net462/​InternalAPI.Shipped.txt Initializes the net462 internal API baseline.
src/​TestFramework/​TestFramework/​InternalAPI/​InternalAPI.Unshipped.txt Records internal run-state API additions.
src/​TestFramework/​TestFramework/​ConfigurationSettings/​TestConfigurationSection.cs Moves the configuration section into core.
src/​TestFramework/​TestFramework/​ConfigurationSettings/​TestConfiguration.cs Moves configuration loading into core.
src/​TestFramework/​TestFramework/​ConfigurationSettings/​DataSourceElementCollection.cs Moves the data-source configuration collection into core.
src/​TestFramework/​TestFramework/​ConfigurationSettings/​DataSourceElement.cs Moves data-source configuration elements into core.
src/​TestFramework/​TestFramework/​ConfigurationSettings/​ConfigurationNames.cs Moves configuration constants into core.
src/​TestFramework/​TestFramework.Extensions/​PublicAPI/​PublicAPI.Unshipped.txt Records common public API removals from Extensions.
src/​TestFramework/​TestFramework.Extensions/​PublicAPI/​net462/​PublicAPI.Unshipped.txt Records .NET Framework public API removals.
src/​TestFramework/​TestFramework.Extensions/​PACKAGE.md Documents new ownership and migration requirements.
src/​TestFramework/​TestFramework.Extensions/​InternalAPI/​net462/​InternalAPI.Unshipped.txt Records .NET Framework internal API removals.
src/​TestFramework/​TestFramework.Extensions/​InternalAPI/​InternalAPI.Unshipped.txt Records internal run-state API removals.
docs/​specifications/​mstest.md Updates the TestRun source link.
docs/​RFCs/​014-TestRun-Current-PlannedTests.md Updates ownership, target coverage, and compatibility documentation.

Review effort: Balanced

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 11, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 11, 2026
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1676331

Triggering check: microsoft.testfx. Analysis is final but inconclusive in part: evidence is flagged incomplete (1 artifact download failure; history extraction budget exhausted, so I can't claim the absence of prior occurrences beyond the 2 history builds that were read).

Failure signatures (Windows legs; Microsoft.Testing.Extensions.UnitTests, net10.0 and net11.0)

  1. TestFxOpenTelemetryTests.RepositoryTestHost_PreservesResultsAndPersistsOnlyWhenEnabled (all data rows: disabled/no-directory/enabled/empty/help/info). The child test host rejects the command line ... --no-ansi --progress off with Unknown option '--no-ansi', so the Assert at TestFxOpenTelemetryTests.cs:467 fails. The test passes an option the test host doesn't register.
  2. TrxInProcessProtocolFixture.cs:173: Assert.ContainsSingle(configuration.EnvironmentVariableProviders) finds 2 providers instead of 1. This affects TrxProtocol_CrashedWithCorruptTail, TrxProtocol_AbortedWithCorruptTail, RegisteredLifecycle_CompletedUnicodeReportPublishesExactArtifactAndSweepsOrphan and RegisteredLifecycle_CanceledHandshakeCanRetryAndCompleteEmptySession.

Assessment (confidence: medium-high that this is PR-related, not flaky)

  • Both signatures are deterministic assertion failures that reproduce on both TFMs. No fail-then-pass retry was proven, so this is not flaky.
  • The same TRX tests passed in the 2 recent main builds whose results were readable (1674754, 1674592); a third history build had no matching results. That points to a regression in this PR's changes or in how the tests are set up, not a pre-existing durable defect.
  • No crash or hang diagnostics. No slowness regression met the threshold. No existing open or closed issue matches these signatures, and no issue was created because the durable-issue threshold (recurrence on main or unrelated commits) isn't met.

Next step: check whether the PR registers an additional environment-variable provider, which would break the exact-count assertion in TrxInProcessProtocolFixture. Also check why the --no-ansi option isn't recognised by the child host used in TestFxOpenTelemetryTests: the extension or option registration may be missing from that test host.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~39 AIC · ⌖ ~1 AIC · ⊞ 10.1K

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants