diff --git a/docs/decisions/explicit-mstest-packages.md b/docs/decisions/explicit-mstest-packages.md index 266b21a9f7..c746f0860b 100644 --- a/docs/decisions/explicit-mstest-packages.md +++ b/docs/decisions/explicit-mstest-packages.md @@ -378,3 +378,319 @@ The known native net462 PlatformServices System.Reflection.Metadata failure has only static baseline attribution and was not repaired or experimentally chased. The coordinator retains downstream rollout/servicing ownership; layers 3 and 4 retain their explicitly deferred policy/native-package responsibilities. + +## Native parity slice: layer 4 + +2026-11-13: carve only the residual native rules, probes and validators from +`00428d48c212cf3630dcf6fa31d131c5ae745c98` onto the verified SDK parent +`17b8abac7c8959e73413e01f2c92b0998b590551`. Shipping source, package policy, +runtime behavior and the lower layers' evidence remain unchanged. + +Native modern `MSTest.Uwp` and `MSTest.WinUI` archives must use canonical +`lib`/`buildTransitive` paths, matching adapters/services/satellites, and an exact +`[matching core version]` dependency without ordinary/other-flavor payload leaks. +Desktop and both UI library consumers must load the same ordinary core bytes. +An absent ordinary-core target-framework attribute remains supported; declared +wrong/Windows metadata is rejected, with hash comparisons keyed by the planned +TFM. Reuse the narrow UI-only range-tool mode, not a new metadata producer. +Native classic UAP production remains unsupported. + +Keeping flavored cores or copying the whole unsplit PR would violate this +boundary. The chosen native delta follows the actual source inputs, preserves +Hosting/core alignment and append-only evidence, and updates portable fixtures +only where required by the existing `net10.0;net11.0` contract. SDK test scheduling +stays method-level/CPU-count, with per-test temporary paths and bounded joined +child processes; shared producer inputs remain read-only. + +Planned evidence is `//:sdk_package_tests`, the five staged native Windows library +probes plus targeting/asset tests, and the two modern UI archive targets checked +with the maintained `Assert-PackageLayout` logic. These are nonvisual library and +archive contracts, not UI activation or full ordinary native-consumer proof. +The known native net462 PlatformServices metadata-reference failure, CI, +other architectures, classic UAP and release-branch validation remain explicit +gaps; rollout/servicing ownership stays with the series coordinator. + +Executed native evidence: the 19 native inputs and patch remained byte-identical +before/after all runs. The native patch SHA-256 against the parent above is +`8A5A3F20DCE2FEBA1FE187D43C574013B5B108ED9CBFEE5304C7C02D090E35C1`; +`src/`, the Hosting/core rules and producer patches were not changed. Bazel +`9.1.1` used pinned SDK `11.0.100-rc.3.26522.148` on Windows x64. + +| Requirement | Command / executed evidence | +| --- | --- | +| SDK/meta-package contracts | `eng\bazel.ps1 test //:sdk_package_tests --jobs=2 --test_output=errors`: exit **0**, TRX **15 passed, 0 failed, 0 skipped**. `Generate_SdkPreservesTemplatesAndComponentVersions`, `Generate_MetaPackagePreservesDependencyAndAssetFlow` and all 13 `Generate_RejectsContractDriftBeforeWritingOutputs` rows executed, including `meta-retired-framework`, `meta-ui-reference`, and `meta-ui-condition`. | +| Independent SDK discovery | `eng\bazel.ps1 run //:sdk_package_tests_app --noshow_progress '--' --list-tests`: exit **0**, the same 15 source-planned identities. Direct `.bat` discovery lacked runfiles; the test wrapper also rejects discovery with its execution-only minimum/TRX defaults. The application target is the correct discovery entry. | +| Shared ordinary core and isolated UI libraries | `eng\bazel\windows\validate.ps1 -ArtifactsDirectory artifacts\validation\native-parity-e4c16b\windows`: exit **0**; five staged consumers (`windows_probe_packaged_net10`, `windows_probe_packaged_net11`, `windows_probe_desktop`, `windows_probe_uwp`, `windows_probe_winui`) and both executed XML cases (`windows_targeting_test`, `windows_asset_selection_test`) passed. Runtime versions were `10.0.14` and `11.0.0`. | +| Actual modern UI archives | `eng\bazel.ps1 build //:mstest_uwp_package //:mstest_winui_package --jobs=2 --noshow_progress --show_result=0`: exit **0**. AST-loaded maintained `Assert-PackageLayout` checks passed for those two cquery-resolved archives; DLLs/satellites matched the staged native probes and build assets matched current source. This was scoped UI validation, not a full validator/ordinary consumer run. | + +Both UI packages are `5.0.0-preview.bazel` and declare exactly +`[5.0.0-preview.bazel]` for core. UWP has 43 entries/26 satellites; WinUI has +56 entries/39 satellites. Their archive SHA-256 values are respectively +`1F518CFD8CDF3B02B8667625FBBAB4C8E3F72CA62D0584773E95F6C0504DC248` and +`17BEE8E98003EAF43C6EFB373CC1CDCEA36F748E931993F80B71F6783EF08BB4`. +Desktop/UWP/WinUI loaded ordinary core SHA-256 +`D392BF09B480D19856C94B7DD47C092CA348B5B89168152D0E45FDCCE93873B1`, +with no declared target-framework/Windows metadata. + +**Test-value evidence:** replacing each exact core pin with a compatible range +in owned archive copies triggered the maintained exact-pin assertion twice. +Injecting the actual UWP attribute DLL into an owned WinUI publish copy made the +unchanged `LibraryProbe` reject the other flavor (exit **-532462766**); removing +only that injected file restored exit **0** with the same core hash. These are +executed output/layout mutations, not pristine-baseline runs. The earlier +flavored-core configuration and SDK guard removals were not executed as +counterfactuals. Logs, TRX/XML, input hashes and package/probe manifests are +retained under the task-owned `artifacts\validation\native-parity-e4c16b`. + +The full native closure was deliberately not rerun: the previously observed +net462 PlatformServices `CS0234` for missing `System.Reflection.Metadata` / +`PortableExecutable` references remains outside scope. Its prerequisite +attribution is static, not an executed pristine-baseline proof. CI, non-x64, +classic UAP production, real UI activation and release-branch proof remain +unavailable; no downstream enablement or backport is made. + +Owned mutation layouts and bad archive copies were removed. The isolated Bazel +server (PID `92900`) shut down with exit **0**; no app registration, shared-cache +cleanup or other-worktree cleanup was performed. + +Before publication, integrated SDK parent +`06ee2b01d972423dd9d4f06b8ef5c8256cda9e3c`. Its only change from the tested +parent was the SDK decision/evidence document; native/product inputs and the +native patch hash above are unchanged. + +### Maintenance alignment: 2026-11-16 + +The current-main binary-compatibility contract supersedes the earlier +Extensions-DLL absence checks. Verified package-layer commit +`a9406f988adb7cab787bddda02431ddfc232f948` contains the nonpackable +`TestFramework.Compatibility` project. It emits the +`MSTest.TestFramework.Extensions` facade for the ordinary core TFMs and references +core in one direction only. Core's package owns its same-TFM facade DLL/XML; +there is no separate facade package, dependency, UI implementation or UI type +forwarder. The old implementation/project/package owner remains retired. + +Native parity must preserve that concrete contract rather than drop the facade +or retain Windows-flavored core/facade copies. The native core package carries +ordinary core plus its facade; modern UI packages depend on that core package. +Desktop/UWP/WinUI probes must resolve both shared assemblies from the ordinary +native producers. `TestContext` and `DeploymentItemAttribute` still resolve to +one actual core definition through the facade. UI attributes remain exclusively +in the selected UI assembly. + +| Requirement | Focused evidence to rerun after the verified SDK parent arrives | +| --- | --- | +| Same-TFM facade payload owned only by core | `Assert-PackageLayout` and the real native package consumers, including restored/cached/loaded hashes for the facade. | +| Legacy non-UI names resolve to the one core definition | `NativePackageTests.PackagedTargetsProvideCoreAndParallelization` and `LibraryProbe.VerifyFrameworkApi`; reject public definitions or UI forwarding in the facade. | +| No flavored core/facade or other-UI leakage | `windows_targeting_test`, the five staged Windows consumers, and per-assembly/TFM hash comparisons. | +| First-use provenance and native net462 build | Missing/existing evidence-index regression; pinned net462 PlatformServices and adapter builds. | + +Preparation reproduced the native net462 metadata-reference failure and repaired +its missing `System.Reflection.Metadata` dependency. The supplied package commit +also contains that dependency, so integration must preserve the exact net462 +selection without adding a duplicate. The first-use provenance regression fails +with the old unguarded read and passes with the matching `Test-Path` guard. +These are preparation results, not final rebased-head or remote-CI evidence. +Rebase and publication remain gated on the coordinator's verified final SDK +parent. Launcher ownership, activation, cancellation and result/exit semantics +are unchanged; the facade checks are nonvisual. + +The same two-case layout harness executed the actual `Assert-PackageLayout` +function before and after the adaptation. `CoreOwnsSameFrameworkFacade` fails +against the old source blob `42c5a5e6547a66ff36cc191bf083dfae9fcb46fe` because it +rejects core's facade as a retired payload; the adapted function accepts it. +`CoreRejectsMissingFacade` then verifies the precise missing same-TFM DLL error. +The old run exits 1 and the adapted run exits 0 with both cases passing. These +archives contain synthetic entry bytes: this proves the layout validator's +contract, not CLR type forwarding, real package provenance or final CI. + +Interim execution at `ebc0578dab5bc5e97f8f0965fd53d718218a2cdf`, based on released +SDK parent `848e85672405fb84c0526048dd8d225bc883b4b3`, passed the maintained native +MSTest validator with ten real archives, fresh net10/net11 consumers, MSBuild +invocation and satellite removal/restoration. The same compiled +`PackagedTargetsProvideCoreAndParallelization` test also passed, failed its +explicit facade-presence assertion after removing only the DLL from an owned +publish copy, and passed after restoration: one executed case and exits 0/2/0. +Windows validation passed all five CLR probes, exact 26/5 compatibility +inventories and four additional unit suites (1,096 passed, 13 existing skips). +The full native graph and composed-product validator also passed on Windows. +These results do not claim final repaired-stack CI. + +The native SDK validator exposed a stale expectation in this layer's four new +UI evaluation cases: packaged UWP/WinUI test applications correctly receive +`Microsoft.Testing.Extensions.PackagedApp.MSBuild` from the shipping SDK's +`Runner/Common.targets`. Preserve that helper and its aligned MTP version in +the expected package set rather than disabling the existing SDK behavior. +After that expectation-only correction, the native SDK/meta-package validator +passed all fourteen evaluations, five real consumer builds, default/minimal SDK +runner checks, `dotnet test`, and VSTest execution, including their expected +failure and empty-selection exits. UI evaluation remains policy coverage, not +native XAML application compilation or activation. + +### Final native validation: 2026-11-16 + +Real package/SDK/controller validation was repeated at +`ad6ebd567cfe46e7f3f85de00ea3c8d760cecebb` on repaired SDK parent +`a84c52f49168d02ca50b6335f7c8eb2132aa2d77`, not borrowed from the interim +parent. Final implementation `abf2b526d70c2b1dd938cfd8c5b3876bb27e285a` +replays unchanged onto SDK `619c3129e23a478b480a6e401ab61fbfc662f7f2`. +The only difference between those tested and publication trees is the +unconsumed mutation catalog. Their `src` and `test` trees, native rules, +scripts, fixtures and documentation were verified identical before this +evidence-only addition. The inherited exact-identity, 26/5 inventory, net462 +metadata dependency and lower OpenTelemetry environment repairs remain intact. + +Bazel is `9.1.1`; its pinned SDK/runtime are +`11.0.100-rc.3.26522.148` / `11.0.0-rc.3.26522.148`. Consumer publishes run their +actual self-contained net10/net11 runtimes. Output/server and same-version +consumer feeds/caches are task-owned. + +| Exact command | Observed result | +| --- | --- | +| `pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-mstest-packages.ps1 -OutputRoot Q:\tfb\n12451-80f9f2\m-final -IncludeWindowsPlatformPackages -VerifySatelliteSensitivity` | Exit 0; ten real archives, fresh net10/net11 and MSBuild consumers, exact loaded/cache hashes, intended failure/empty exits and satellite removal/restoration. | +| `pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-sdk-packages.ps1 -EvidenceRoot Q:\tfb\n12451-80f9f2\s-final -Jobs 2` | Exit 0; all fourteen package-policy evaluations, five consumer builds, real SDK/default/minimal and metapackage execution, native `dotnet test`, and expected negative exits. | +| `pwsh -NoLogo -NoProfile -File .\eng\bazel\windows\validate-app-model.ps1 -ArtifactsDirectory Q:\tfb\n12451-80f9f2\a-final` | Exit 0; real registration/activation, suppressed launcher, both net10/net11 controllers, discovery without handshake, failure/zero-result paths and sixteen append-only provenance records. The owned registration was removed. | +| `.\eng\bazel.ps1 test //:sdk_package_tests //:build_tools_tests //:windows_targeting_test //:windows_asset_selection_test --jobs=2 --nocache_test_results --test_output=errors --noshow_progress --show_result=0` | Exit 0; 15 SDK cases and 13 build-tool cases executed/passed with no skips, plus both executed analysis cases. | +| `python .github\scripts\test_mutation_testing.py` and `python .github\scripts\mutation_testing.py matrix --manifest eng\mutation-testing\modules.json --repository-root .` | Both exit 0; 37 catalog tests, 43 source projects = 32 enabled modules + 11 explicit exclusions. | +| `markdownlint-cli2 docs\decisions\explicit-mstest-packages.md` | Exit 0; the touched record passes the existing repository rules. | + +The facade shipping-layout experiment was repeated against the final freshly +published net11 consumer, keeping its compiled test DLL unchanged: +`PackagedTargetsProvideCoreAndParallelization` executes once and passes with +the facade, fails its explicit facade-presence assertion after deleting only +that DLL from an owned copy (exit 2), and passes after restoration. The paired +TRX files and input hashes are retained. The first-use provenance guard also +passes both missing/existing-index cases; its old unguarded source fails the +missing-index case. These are controlled input/layout experiments, not claims +that a pristine parent or a particular CI thread interleaving was executed. + +Final raw logs/binlogs/TRX, package and loaded-bit manifests are retained under +the task-owned output root. The app-model validator's existing null revision +field is supplemented by a source-state record and matching captured input +hashes. Evidence paths are local, not public artifact URLs. Native classic UAP, +MSTest XAML/UI-thread application activation, other architectures and release +branches remain unclaimed; the real activation above uses the native console +app-model host. No rollout/backport or merge action is performed. + +### Native adapter package support boundary: 2026-11-16 + +Review on `204d4850de60b2b2fbc28a4a1934ad73f2b2c673` identified an accidental +Windows-only `net462` adapter package group. The native adapter's first-party +dependency packages expose only `net10.0` and `net11.0` runtime assets. +A real `net462` SDK-style consumer of native `MSTest.TestAdapter` +`5.0.0-preview.bazel` fails restore with exit 1 / `NU1202` at native +`Microsoft.Testing.Extensions.TrxReport.Abstractions` +`3.0.0-preview.bazel`; this is not a net10 proxy or a legacy runtime failure. +The physical nuspecs confirm the same incompatible boundary for native +Telemetry, Platform.MSBuild and Platform. Four resolved native cache archives +match the captured producer hashes; the consumer has a fresh owned cache, +explicit source mapping and no fallback package folders. + +The chosen contract restores the existing portable-only adapter archive: +`net10.0` and `net11.0`, on every producer OS. Remove the misleading legacy +dependency group and payload rather than extending all native MTP packages to +.NET Framework. A full legacy runtime package closure was not designed or +validated here. Keep the existing native net462 compile labels and the +independently consumable ordinary core/facade package unchanged. This does not +change official MSBuild-produced packages or claim native legacy adapter support. + +The maintained package validator requires exactly the two supported adapter +groups and rejects legacy adapter build payloads. The same assertion rejects +the original real archive and passes the freshly repacked archive. A separate +layout mutation removes only the original nuspec's net462 group while retaining +its real payload; the explicit legacy-payload assertion rejects that too. +Each layout probe executes once with zero skips, using the same validator bytes. + +The post-fix legacy adapter restore has an important NuGet distinction: +build-only packages without a `lib` folder restore successfully when no framework +group matches, but contribute no assets. The actual net462 restore exits 0 and +selects only the adapter identity, with no dependencies, build, compile or runtime +assets. It does **not** enable or validate a native legacy adapter runtime. +The initial postprocessing expectation of a direct adapter `NU1202` was wrong; +the real assets were inspected rather than inventing that diagnostic. Adding +marker libraries or unsupported-framework guard assets solely to force it was +not necessary for the existing portable-only archive contract. + +The executed reproduction uses the pinned native SDK +`11.0.100-rc.3.26522.148`, working directory +`Q:\tfb\n12451-80f9f2\net462-review-4587032669\red\consumer`: +`dotnet restore LegacyConsumer.csproj --configfile NuGet.config --packages +Q:\tfb\n12451-80f9f2\net462-review-4587032669\red\confirmed-packages +--disable-build-servers +-bl:Q:\tfb\n12451-80f9f2\net462-review-4587032669\red\restore-confirmed.binlog +--nologo -v:minimal` returned 1. Its source/layout identity, original package +hashes, exact command, NU1202 diagnostic and effective net462 assets/cache are +retained in `restore-evidence.json`, `producer-packages.json`, the copied +nuspecs and restore log beside that binlog. + +The implementation was validated as base +`204d4850de60b2b2fbc28a4a1934ad73f2b2c673` plus the two-file native rules/validator +patch SHA-256 +`86A8AA8C30892EF6C97777D863F8044C4E34F7BBBF9A9632F723C7E5B2BEAA0D`. +The original adapter archive is +`6129E376D454AC01B0EF15E1F6BCE854504AD9F558B6E8875990BA1B4991E50D`; +the corrected archive is +`9652B932323E13E15B4C44B622A07FF642679268D5F59505824E697299A2807D`. +The other nine archive hashes, including core and both UI packages, are unchanged. + +| Requirement / exact check | Executed evidence | +| --- | --- | +| No misleading legacy adapter group / `NativeAdapterPortableLayout` | Actual maintained function fails on the original archive, exit 1, reporting three groups instead of net10/net11; the same function passes the repacked archive, exit 0. | +| No unadvertised legacy payload / `NativeAdapterRejectsLegacyPayloadWithoutLegacyGroup` | Owned original-archive mutation retains real legacy bytes but removes the legacy nuspec group. Exit 1 at the explicit legacy-payload assertion; no setup/compiler failure is counted. | +| Real supported package consumers | `pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-mstest-packages.ps1 -OutputRoot Q:\tfb\n12451-80f9f2\net462-review-4587032669\modern -IncludeWindowsPlatformPackages -VerifySatelliteSensitivity`: exit 0; ten archives, real net10/net11 positive/failure/empty cases, MSBuild parent exits 0/1 and satellite removal/restoration, with new cache/loaded hashes. | +| SDK composition survives the archive boundary | `pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-sdk-packages.ps1 -EvidenceRoot Q:\tfb\n12451-80f9f2\net462-review-4587032669\sdk -Jobs 2`: exit 0; fourteen evaluations, five builds, actual SDK/metapackage runs and native `dotnet test`. | +| Compile labels, Windows CLR ownership and 26/5 inventories remain intact | `pwsh -NoLogo -NoProfile -File .\eng\bazel\windows\validate.ps1 -ArtifactsDirectory Q:\tfb\n12451-80f9f2\net462-review-4587032669\windows`: exit 0; five real CLR probes and four additional unit suites. No registration or XAML activation is claimed. | +| Independently usable native net462 core/facade | Pinned native SDK builds `green\core-consumer\CoreConsumer.csproj` in Release, exit 0; its actual `.NETFramework,Version=v4.6.2` executable exits 0 on CLR `4.0.30319.42000`. Both forwarded identities and archive/cache/loaded core/facade hashes match. | +| Unsupported legacy adapter is not misreported as runtime coverage | `green\legacy-consumer\LegacyConsumer.csproj` restore exits 0 with only the package identity and no selected assets. This is recorded as NuGet's compatible-empty build-package behavior, not a passing legacy adapter scenario. | + +These runs use distinct task-owned feeds/caches and retain commands, binlogs, +TRX, physical nuspecs, manifests and source identities under +`Q:\tfb\n12451-80f9f2\net462-review-4587032669`. The core-only executable and +unsupported-restore records are `green\core-evidence.json` and +`green\legacy-restore-evidence.json`; they include the exact native SDK path, +commands, effective target/cache and loaded paths. The old-archive and mutation +probes are layout evidence, not legacy test execution. No assembly scheduling, +production lifecycle, package versions, official package policy or lower-layer +source changed. + +### UI package isolation guard: 2026-11-16 + +Review on `bb5012718fdeb0d82bdc9eb2cd29fe106bc4e9b7` identified a validator +defect, not a new archive-layout requirement. PowerShell evaluates `-and` and +`-or` at equal precedence, left to right. The combined predicate therefore +allowed a bundled core DLL in either UI archive and a foreign UWP DLL in the +WinUI archive. Parenthesize each flavor-specific conjunction so the existing +shared-core and cross-flavor requirements remain independent. + +Six fixed-input probes use the real native UWP/WinUI archives: two valid controls, +core DLL insertion into each archive, and both cross-flavor DLL insertions. +The four mutations retain each original nuspec and add only one real DLL copied +from the native producer. On the original maintained function, all six execute; +three pass and three fail the expected-rejection assertion, exit 1, with no skips. +`CoreInUwp`, `CoreInWinUI` and `UwpInWinUI` are incorrectly accepted. The reverse +cross-flavor mutation is already rejected and remains a control. + +With explicit conjunction parentheses, the same six archives, assertions and +selection execute and pass, exit 0 with zero skips. Each input archive hash is +identical across the pair. The validator source hashes are +`A5D6E4C6332A26F1DDFC9FE8C9EB7384173FF73EC9067D675299A3B84DE3CE1A` before and +`62334CD8DFF849B02AEE69C9C1C940FB324DD25B331361ED89F206E963E68ACA` after. +The one-file validator patch against the recorded base has SHA-256 +`54674D28C4E71C7CAAA87E4FEB35362104718EDA2E522CAFC0204799D025C03F`. + +`pwsh -NoLogo -NoProfile -File .\eng\bazel\validate-mstest-packages.ps1 +-OutputRoot Q:\tfb\n12451-80f9f2\ui-isolation-review-4588012054\modern +-IncludeWindowsPlatformPackages -VerifySatelliteSensitivity` also returns 0: +ten real archives, fresh net10/net11 positive/failure/empty consumers, MSBuild +parent exits 0/1 and satellite removal/restoration. All ten archive hashes are +unchanged from the preceding adapter-boundary validation, so the existing SDK, +CLR and core-only net462 proofs remain product-equivalent rather than rerun claims. +No compiled product, package version, dependency closure, lifecycle or lower-layer +source changed. + +The fixed-input harness is retained as `regression.ps1`, with `cases.json`, +`before.json`/`after.json`, logs and `evidence-summary.json`, under +`Q:\tfb\n12451-80f9f2\ui-isolation-review-4588012054`. It AST-loads the actual +maintained validator functions, not a copied predicate. These are executed +archive-rejection assertions; the normal package gate separately executes the +real native tests. The bounded mixed-operator check of the layer's other changed +PowerShell validators found only explicitly parenthesized conjunctions. diff --git a/eng/bazel/MSTestPackageVersion.cs b/eng/bazel/MSTestPackageVersion.cs index fccc086b55..2692bad96a 100644 --- a/eng/bazel/MSTestPackageVersion.cs +++ b/eng/bazel/MSTestPackageVersion.cs @@ -7,9 +7,10 @@ using System.Text; using System.Text.Json.Nodes; -if (args.Length != 3) +bool exactFramework = args.Length == 4 && args[3] == "--exact-framework"; +if (args.Length != 3 && !exactFramework) { - throw new ArgumentException("Expected ."); + throw new ArgumentException("Expected [--exact-framework]."); } string version = File.ReadAllText(args[1]).Trim(); @@ -25,9 +26,25 @@ ?? throw new InvalidDataException("Expected a package request object."); if (request["dependencyGroups"] is not JsonArray groups || groups.Count == 0) { - throw new InvalidDataException("Expected nonempty platform dependency groups."); + throw new InvalidDataException("Expected nonempty dependency groups."); } +if (exactFramework) +{ + if (request["metadata"]?["id"]?.GetValue() is not ("MSTest.Uwp" or "MSTest.WinUI")) + { + throw new InvalidDataException("Only UI packages may pin the matching framework dependency."); + } + + string ownVersionFile = request["versionFile"]?.GetValue() + ?? throw new InvalidDataException("The UI package must declare its own version file."); + if (File.ReadAllText(ownVersionFile).Trim() != version) + { + throw new InvalidDataException("The UI package and core dependency versions must match exactly."); + } +} + +string dependencyId = exactFramework ? "MSTest.TestFramework" : "Microsoft.Testing.Platform"; foreach (JsonNode? group in groups) { if (group?["dependencies"] is not JsonArray dependencies) @@ -35,16 +52,16 @@ throw new InvalidDataException("A package dependency group is missing its dependencies."); } - JsonObject[] platform = dependencies.OfType() - .Where(dependency => dependency["id"]?.GetValue() == "Microsoft.Testing.Platform") + JsonObject[] bounded = dependencies.OfType() + .Where(dependency => dependency["id"]?.GetValue() == dependencyId) .ToArray(); - if (platform.Length != 1) + if (bounded.Length != 1) { - throw new InvalidDataException("Each bounded group must declare exactly one platform dependency."); + throw new InvalidDataException($"Each bounded group must declare exactly one {dependencyId} dependency."); } - _ = platform[0].Remove("versionFile"); - platform[0]["version"] = $"[{version}, {nextMajor}.0.0)"; + _ = bounded[0].Remove("versionFile"); + bounded[0]["version"] = exactFramework ? $"[{version}]" : $"[{version}, {nextMajor}.0.0)"; } File.WriteAllText(args[2], request.ToJsonString(), new UTF8Encoding(encoderShouldEmitUTF8Identifier: false)); diff --git a/eng/bazel/mstest-consumer/NativePackageTests.cs b/eng/bazel/mstest-consumer/NativePackageTests.cs index f22f845d0e..5308b08917 100644 --- a/eng/bazel/mstest-consumer/NativePackageTests.cs +++ b/eng/bazel/mstest-consumer/NativePackageTests.cs @@ -48,13 +48,19 @@ public void NativeAssembliesAreLoadedFromConsumer() string evidence = Environment.GetEnvironmentVariable("TESTFX_BAZEL_CONSUMER_EVIDENCE") ?? throw new InvalidOperationException("The consumer evidence path was not supplied."); - File.WriteAllText(evidence, JsonSerializer.Serialize(new + string reportPath = Path.Join(Path.GetDirectoryName(Path.GetFullPath(evidence))!, "loaded-assemblies-" + Guid.NewGuid().ToString("N") + ".json"); + using (FileStream report = File.Open(reportPath, FileMode.CreateNew, FileAccess.Write, FileShare.Read)) { - Runtime = RuntimeInformation.FrameworkDescription, - CoreLib = typeof(object).Assembly.Location, - Framework = expectations.RootElement.GetProperty("Framework").GetString(), - Assemblies = loaded, - })); + JsonSerializer.Serialize(report, new + { + Runtime = RuntimeInformation.FrameworkDescription, + CoreLib = typeof(object).Assembly.Location, + Framework = expectations.RootElement.GetProperty("Framework").GetString(), + Assemblies = loaded, + }); + } + + File.AppendAllLines(evidence, [reportPath]); } [TestMethod] @@ -80,9 +86,31 @@ public void KoreanSatellitesAreCopiedAndLoadable() } [TestMethod] - public void PackagedTargetsProvideExtensionsAndParallelization() + public void PackagedTargetsProvideCoreAndParallelization() { Assert.AreEqual("MSTest.TestFramework", typeof(TestContext).Assembly.GetName().Name); + Assert.AreSame(typeof(Assert).Assembly, typeof(TestContext).Assembly); + Assert.AreSame(typeof(Assert).Assembly, typeof(DeploymentItemAttribute).Assembly); + Assert.IsTrue(File.Exists(Path.Join(AppContext.BaseDirectory, "MSTest.TestFramework.Extensions.dll")), "The core package must provide the native compatibility facade."); + Assembly compatibility = Assembly.Load("MSTest.TestFramework.Extensions"); + Assert.AreSame(typeof(TestContext), compatibility.GetType(typeof(TestContext).FullName!)); + Assert.AreSame(typeof(DeploymentItemAttribute), compatibility.GetType(typeof(DeploymentItemAttribute).FullName!)); + Assert.IsFalse(compatibility.DefinedTypes.Any(type => type.IsPublic || type.IsNestedPublic)); + Assert.IsTrue(compatibility.GetForwardedTypes().All(type => type.Assembly == typeof(Assert).Assembly)); + string[] directories = + [ + nameof(TestContext.TestRunDirectory), + nameof(TestContext.DeploymentDirectory), + nameof(TestContext.ResultsDirectory), + nameof(TestContext.TestRunResultsDirectory), + nameof(TestContext.TestResultsDirectory), + nameof(TestContext.TestTempDirectory), + ]; + foreach (string directory in directories) + { + Assert.IsNotNull(typeof(TestContext).GetProperty(directory), directory); + } + ParallelizeAttribute? parallelize = typeof(NativePackageTests).Assembly.GetCustomAttribute(); Assert.IsNotNull(parallelize); Assert.AreEqual(ExecutionScope.MethodLevel, parallelize.Scope); diff --git a/eng/bazel/mstest-packages.bzl b/eng/bazel/mstest-packages.bzl index 8362f2c35d..136b661e5b 100644 --- a/eng/bazel/mstest-packages.bzl +++ b/eng/bazel/mstest-packages.bzl @@ -1,4 +1,4 @@ -"""The portable MSTest package layouts, with explicitly owned bundled payloads.""" +"""MSTest package layouts, with explicitly owned core and platform payloads.""" load( "@rules_dotnet//dotnet/private:providers.bzl", @@ -7,9 +7,15 @@ load( "NuGetPackInfo", ) load("@rules_dotnet//dotnet/private/rules/nuget_pack:layout.bzl", "extra_files_layout") -load(":defs.bzl", "PORTABLE_FRAMEWORKS") +load(":defs.bzl", "PORTABLE_FRAMEWORKS", "STANDARD_FRAMEWORKS") load(":packages.bzl", "package_build_assets", "package_version") +def mstest_package_tfm(framework): + """Match NuGet's Windows folder normalization without shortening classic UAP.""" + if "-windows" in framework and len(framework.split("-windows")[1].split(".")) == 4: + return framework.removesuffix(".0") + return framework + def _frameworks_impl(_settings, attr): return { framework: {"@rules_dotnet//dotnet:target_framework": framework} @@ -50,6 +56,7 @@ def _package_impl(ctx): groups = [] version_files = [ctx.file.version_file] platform_version_file = None + core_version_file = None dependencies = [] for target in ctx.attr.deps: package = target[NuGetPackInfo] @@ -60,6 +67,10 @@ def _package_impl(ctx): if not version_file: fail("The native platform dependency must carry its generated version file.") platform_version_file = version_file + if ctx.attr.exact_core_dependency and package.package_id == "MSTest.TestFramework": + if not version_file: + fail("The native core dependency must carry its generated version file.") + core_version_file = version_file dependencies.append(struct( id = package.package_id, version = package.version or "", @@ -69,6 +80,10 @@ def _package_impl(ctx): if ctx.attr.bound_platform and not platform_version_file: fail("A bounded package must declare the native platform dependency.") + if ctx.attr.exact_core_dependency and not core_version_file: + fail("An exact core package must declare the native framework dependency.") + if core_version_file and platform_version_file: + fail("Core pinning and platform bounds must use separate package contracts.") if len(ctx.attr.directories) != len(ctx.attr.resource_directories): fail("Each assembly directory needs an explicit satellite-resource policy.") for framework in ctx.attr.target_frameworks: @@ -76,12 +91,13 @@ def _package_impl(ctx): if len(targets) != len(ctx.attr.directories): fail("{} is missing native assembly outputs for {}".format(ctx.label, framework)) for index, target in enumerate(targets): - directory = ctx.attr.directories[index].replace("{tfm}", framework) + package_tfm = mstest_package_tfm(framework) + directory = ctx.attr.directories[index].replace("{tfm}", package_tfm) resource_directory = ctx.attr.resource_directories[index] - # Adapter satellites are shared once, using the lowest portable TFM. + # Ordinary adapter satellites are shared once, from its first portable TFM. if "{tfm}" not in resource_directory and framework != ctx.attr.target_frameworks[0]: resource_directory = "" - name = _add_assembly(entries, target, framework, directory, resource_directory.replace("{tfm}", framework)) + name = _add_assembly(entries, target, framework, directory, resource_directory.replace("{tfm}", package_tfm)) names[name] = True external_dependencies = [] @@ -92,7 +108,7 @@ def _package_impl(ctx): if info.nuget_info == None: fail("{} is not an external NuGet dependency".format(target.label)) external_dependencies.append(struct(id = info.name, version = info.version, exclude = ["Build", "Analyzers"])) - groups.append(struct(targetFramework = framework, dependencies = dependencies + external_dependencies)) + groups.append(struct(targetFramework = mstest_package_tfm(framework), dependencies = dependencies + external_dependencies)) for target in ctx.attr.task_libraries: _add_assembly(entries, target, "netstandard2.0", "_MSBuildTasks/netstandard2.0", "_MSBuildTasks/netstandard2.0") @@ -115,7 +131,8 @@ def _package_impl(ctx): nupkg = ctx.actions.declare_file(ctx.label.name + "/" + ctx.label.name + ".nupkg") nuspec = ctx.actions.declare_file(ctx.label.name + "/" + ctx.label.name + ".nuspec") request = ctx.actions.declare_file(ctx.label.name + "/nuget_pack_request.json") - template = ctx.actions.declare_file(ctx.label.name + "/nuget_pack_request.template.json") if platform_version_file else request + range_version_file = core_version_file or platform_version_file + template = ctx.actions.declare_file(ctx.label.name + "/nuget_pack_request.template.json") if range_version_file else request ctx.actions.write(template, json.encode(struct( output = nupkg.path, nuspecOutput = nuspec.path, @@ -131,13 +148,18 @@ def _package_impl(ctx): dependencyGroups = groups, files = [struct(source = by_path[path].path, target = path) for path in sorted(by_path.keys())], ))) - if platform_version_file: + if range_version_file: + arguments = [template.path, range_version_file.path, request.path] + inputs = [template, range_version_file] + if core_version_file: + arguments.append("--exact-framework") + inputs.append(ctx.file.version_file) ctx.actions.run( executable = ctx.executable._range_tool, - arguments = [template.path, platform_version_file.path, request.path], - inputs = [template, platform_version_file], + arguments = arguments, + inputs = depset(inputs), outputs = [request], - mnemonic = "TestFxPlatformDependencyRange", + mnemonic = "TestFxFrameworkDependencyRange" if core_version_file else "TestFxPlatformDependencyRange", ) ctx.actions.run( executable = ctx.attr._packer[DefaultInfo].files_to_run, @@ -175,6 +197,7 @@ _package = rule( "dependency_excludes": attr.string_list_dict(), "nuget_deps": attr.label_list(providers = [DotnetAssemblyRuntimeInfo], cfg = _frameworks), "bound_platform": attr.bool(), + "exact_core_dependency": attr.bool(), "package_id": attr.string(mandatory = True), "description": attr.string(mandatory = True), "readme": attr.label(allow_single_file = True, mandatory = True), @@ -191,7 +214,7 @@ def _asset(files, name, source, destination): files[":" + name] = destination def native_mstest_packages(): - """Declare portable packages using the explicit compatibility task wrappers.""" + """Declare ordinary packages and opt-in modern Windows platform packages.""" package_version(name = "mstest_package_version", property_name = "VersionPrefix") framework_project = "src/TestFramework/TestFramework" adapter_project = "src/Adapter/MSTest.TestAdapter" @@ -212,6 +235,7 @@ def native_mstest_packages(): description = "Provides the libraries for writing tests with MSTest. MSTest is Microsoft supported Test Framework.", readme = framework_project + "/PACKAGE.md", libraries = [":framework", ":framework_compatibility"], + target_frameworks = STANDARD_FRAMEWORKS + ["net462"], directories = ["lib/{tfm}", "lib/{tfm}"], resource_directories = ["lib/{tfm}", ""], files = framework_files, @@ -256,6 +280,7 @@ def native_mstest_packages(): description = "Discovers and runs MSTest tests using VSTest or Microsoft Testing Platform. MSTest is Microsoft supported Test Framework.", readme = adapter_project + "/PACKAGE.md", libraries = [":adapter", ":platform_services"], + target_frameworks = PORTABLE_FRAMEWORKS, directories = ["buildTransitive/{tfm}", "buildTransitive/{tfm}"], resource_directories = ["buildTransitive/_localization", "buildTransitive/_localization"], files = adapter_files, @@ -273,3 +298,39 @@ def native_mstest_packages(): name = "mstest_package_closure", srcs = [":mstest_packages", ":packages", ":analyzers_package"], ) + + platform_packages = [] + for flavor, package_id, framework in [ + ("uwp", "MSTest.Uwp", "net10.0-windows10.0.17763.0"), + ("winui", "MSTest.WinUI", "net10.0-windows10.0.18362.0"), + ]: + name = "mstest_" + flavor + "_package" + project = "src/TestFramework/" + package_id + tfm = mstest_package_tfm(framework) + files = { + adapter_project + "/buildTransitive/common/MSTest.TestAdapter.targets": "buildTransitive/" + tfm + "/" + package_id + ".targets", + adapter_project + "/buildTransitive/Parallelize.targets": "buildTransitive/" + tfm + "/Parallelize.targets", + "src/TestFramework/PlatformPackage/MSTest.Platform.props": "buildTransitive/" + tfm + "/" + package_id + ".props", + } + _package( + name = name, + package_id = package_id, + version_file = ":mstest_package_version", + description = "Provides UI-thread test attributes and matching MSTest adapters for " + package_id + ".", + readme = project + "/PACKAGE.md", + libraries = [":mstest_" + flavor, ":adapter_" + flavor, ":platform_services_" + flavor], + target_frameworks = [framework], + directories = ["lib/{tfm}", "buildTransitive/{tfm}", "buildTransitive/{tfm}"], + resource_directories = ["lib/{tfm}", "buildTransitive/{tfm}/_localization", "buildTransitive/{tfm}/_localization"], + files = files, + deps = [":framework_package", ":trx_abstractions_package", ":telemetry_package", ":platform_msbuild_package"], + exact_core_dependency = True, + nuget_deps = ["@paket.source//microsoft.testplatform.objectmodel"], + target_compatible_with = ["@platforms//os:windows"], + ) + platform_packages.append(":" + name) + native.filegroup( + name = "mstest_platform_package_closure", + srcs = [":mstest_package_closure"] + platform_packages, + target_compatible_with = ["@platforms//os:windows"], + ) diff --git a/eng/bazel/sdk-packages.bzl b/eng/bazel/sdk-packages.bzl index 5e2749c2e6..ebca496f80 100644 --- a/eng/bazel/sdk-packages.bzl +++ b/eng/bazel/sdk-packages.bzl @@ -143,6 +143,6 @@ def sdk_packages(): testfx_test( name = "sdk_package_tests", binary = ":sdk_package_tests_app", - args = ["--minimum-expected-tests", "12"], + args = ["--minimum-expected-tests", "15"], size = "medium", ) diff --git a/eng/bazel/validate-mstest-packages.ps1 b/eng/bazel/validate-mstest-packages.ps1 index 43e3125462..f2e293691e 100644 --- a/eng/bazel/validate-mstest-packages.ps1 +++ b/eng/bazel/validate-mstest-packages.ps1 @@ -6,11 +6,13 @@ [CmdletBinding()] param( [string] $OutputRoot, - [switch] $VerifySatelliteSensitivity + [switch] $VerifySatelliteSensitivity, + [switch] $IncludeWindowsPlatformPackages ) $ErrorActionPreference = 'Stop' $PSNativeCommandUseErrorActionPreference = $false +if ($IncludeWindowsPlatformPackages -and -not $IsWindows) { throw 'Native UI package validation requires Windows.' } $root = Split-Path (Split-Path $PSScriptRoot -Parent) -Parent $bazel = Join-Path $root 'eng\bazel.ps1' if (-not $OutputRoot) { $OutputRoot = Join-Path $root 'artifacts\bazel-mstest-validation' } @@ -18,13 +20,20 @@ $directory = Join-Path ([System.IO.Path]::GetFullPath($OutputRoot)) ([DateTime]: $feed = Join-Path $directory 'feed' New-Item -ItemType Directory -Path $feed -Force | Out-Null $frameworks = @('net10.0', 'net11.0') -$expectedIds = @( +$coreFrameworks = @('netstandard2.0', 'net462') + $frameworks +$platformFrameworks = @{ + 'MSTest.Uwp' = 'net10.0-windows10.0.17763' + 'MSTest.WinUI' = 'net10.0-windows10.0.18362' +} +$portableIds = @( 'MSTest.TestFramework', 'MSTest.TestAdapter', 'MSTest.Analyzers', 'Microsoft.Testing.Platform', 'Microsoft.Testing.Platform.MSBuild', 'Microsoft.Testing.Extensions.Telemetry', 'Microsoft.Testing.Extensions.TrxReport', 'Microsoft.Testing.Extensions.TrxReport.Abstractions' ) -$passingTests = @('NativeAssembliesAreLoadedFromConsumer', 'KoreanSatellitesAreCopiedAndLoadable', 'PackagedTargetsProvideExtensionsAndParallelization') +$expectedIds = $portableIds +if ($IncludeWindowsPlatformPackages) { $expectedIds += @('MSTest.Uwp', 'MSTest.WinUI') } +$passingTests = @('NativeAssembliesAreLoadedFromConsumer', 'KoreanSatellitesAreCopiedAndLoadable', 'PackagedTargetsProvideCoreAndParallelization') $packages = @{} $steps = [System.Collections.Generic.List[object]]::new() $runs = [System.Collections.Generic.List[object]]::new() @@ -89,8 +98,14 @@ function Assert-Trx([string] $Path, [string[]] $Expected, [int] $Failed = 0, [st function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { $groups = @($Metadata.SelectNodes('./*[local-name()="dependencies"]/*[local-name()="group"]')) - if ($Id -in @('MSTest.TestFramework', 'MSTest.TestAdapter', 'Microsoft.Testing.Platform.MSBuild', 'Microsoft.Testing.Extensions.Telemetry')) { - Assert-Names $frameworks @($groups | ForEach-Object targetFramework) "$Id framework groups" + $packageFrameworks = switch ($Id) { + 'MSTest.TestFramework' { $coreFrameworks } + 'MSTest.TestAdapter' { $frameworks } + { $platformFrameworks.ContainsKey($_) } { @($platformFrameworks[$Id]) } + default { $frameworks } + } + if ($Id -in @('MSTest.TestFramework', 'MSTest.TestAdapter', 'Microsoft.Testing.Platform.MSBuild', 'Microsoft.Testing.Extensions.Telemetry', 'MSTest.Uwp', 'MSTest.WinUI')) { + Assert-Names $packageFrameworks @($groups | ForEach-Object targetFramework) "$Id framework groups" } $allDependencies = @($Metadata.SelectNodes('.//*[local-name()="dependency"]')) foreach ($dependency in $allDependencies) { @@ -98,7 +113,10 @@ function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { throw "$Id declares a private bundled assembly as package dependency: $($dependency.id)" } } - foreach ($framework in $frameworks) { + if ($Id -ne 'MSTest.TestFramework' -and @($Archive.Entries | Where-Object Name -Like 'MSTest.TestFramework.Extensions.*').Count) { + throw "$Id must obtain the compatibility facade from the shared core package." + } + foreach ($framework in $packageFrameworks) { switch ($Id) { 'MSTest.TestFramework' { foreach ($name in @('MSTest.TestFramework', 'MSTest.TestFramework.Extensions')) { @@ -119,6 +137,24 @@ function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { $null = Get-ZipHash $Archive "buildTransitive/$framework/Parallelize.targets" if (@($Archive.Entries | Where-Object FullName -Like 'lib/*').Count) { throw 'The adapter must not expose a lib folder.' } } + { $platformFrameworks.ContainsKey($_) } { + foreach ($name in @('MSTest.TestAdapter', 'MSTestAdapter.PlatformServices')) { + $null = Get-ZipHash $Archive "buildTransitive/$framework/$name.dll" + } + $null = Get-ZipHash $Archive "lib/$framework/$Id.dll" + foreach ($extension in @('props', 'targets')) { + $null = Get-ZipHash $Archive "buildTransitive/$framework/$Id.$extension" + } + $null = Get-ZipHash $Archive "buildTransitive/$framework/Parallelize.targets" + if (@($Archive.Entries | Where-Object FullName -Like 'buildTransitive/common/*').Count) { + throw "$Id must use package-local targets, not a shared wrapper directory." + } + if (@($Archive.Entries | Where-Object { $_.FullName -like '*/MSTest.TestFramework.dll' -or + ($_.FullName -like 'lib/*/MSTest.Uwp.dll' -and $Id -ne 'MSTest.Uwp') -or + ($_.FullName -like 'lib/*/MSTest.WinUI.dll' -and $Id -ne 'MSTest.WinUI') }).Count) { + throw "$Id must depend on shared core and isolate its own UI assembly." + } + } 'Microsoft.Testing.Platform.MSBuild' { $null = Get-ZipHash $Archive "lib/$framework/Microsoft.Testing.Extensions.MSBuild.dll" $null = Get-ZipHash $Archive '_MSBuildTasks/netstandard2.0/Microsoft.Testing.Platform.MSBuild.dll' @@ -145,12 +181,20 @@ function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { } $catalogs = switch ($Id) { 'MSTest.TestFramework' { - @{ Directory = 'src\TestFramework\TestFramework\Resources\xlf'; Stem = 'FrameworkMessages'; Assembly = 'MSTest.TestFramework'; Folders = @($frameworks | ForEach-Object { "lib/$_" }) } + @{ Directory = 'src\TestFramework\TestFramework\Resources\xlf'; Stem = 'FrameworkMessages'; Assembly = 'MSTest.TestFramework'; Folders = @($coreFrameworks | ForEach-Object { "lib/$_" }) } } 'MSTest.TestAdapter' { @{ Directory = 'src\Adapter\MSTest.TestAdapter\Resources\xlf'; Stem = 'PlatformAdapterResources'; Assembly = 'MSTest.TestAdapter'; Folders = @('buildTransitive/_localization') } @{ Directory = 'src\Adapter\MSTestAdapter.PlatformServices\Resources\xlf'; Stem = 'Resource'; Assembly = 'MSTestAdapter.PlatformServices'; Folders = @('buildTransitive/_localization') } } + { $platformFrameworks.ContainsKey($_) } { + $folder = "buildTransitive/$($platformFrameworks[$Id])/_localization" + @{ Directory = 'src\Adapter\MSTest.TestAdapter\Resources\xlf'; Stem = 'PlatformAdapterResources'; Assembly = 'MSTest.TestAdapter'; Folders = @($folder) } + @{ Directory = 'src\Adapter\MSTestAdapter.PlatformServices\Resources\xlf'; Stem = 'Resource'; Assembly = 'MSTestAdapter.PlatformServices'; Folders = @($folder) } + } + 'MSTest.WinUI' { + @{ Directory = 'src\TestFramework\MSTest.WinUI\Resources\xlf'; Stem = 'FrameworkExtensionsMessages'; Assembly = 'MSTest.WinUI'; Folders = @("lib/$($platformFrameworks[$Id])") } + } } foreach ($catalog in $catalogs) { foreach ($source in Get-ChildItem -LiteralPath (Join-Path $root $catalog.Directory) -Filter "$($catalog.Stem).*.xlf" -File) { @@ -158,8 +202,8 @@ function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { foreach ($folder in $catalog.Folders) { $null = Get-ZipHash $Archive "$folder/$culture/$($catalog.Assembly).resources.dll" } } } - if ($Id -eq 'MSTest.TestFramework' -and @($Archive.Entries | Where-Object FullName -Like '*MSTest.TestFramework.Extensions.resources.dll').Count) { - throw 'The shipping framework layout bundles only the core satellite resources.' + if ($Id -eq 'MSTest.TestAdapter' -and @($Archive.Entries | Where-Object FullName -Match '(^|/)(net4\d+|winui|uwp|uap[^/]*|net\d+\.\d+-windows[^/]*)(/|$)').Count) { + throw 'The ordinary adapter must not ship legacy or UI flavor payloads or build assets.' } foreach ($group in $groups) { $dependencies = @($group.SelectNodes('./*[local-name()="dependency"]')) @@ -170,11 +214,17 @@ function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { throw 'The framework must retain native analyzer/build assets and exclude only compile/runtime.' } } - 'MSTest.TestAdapter' { + { $_ -in @('MSTest.TestAdapter', 'MSTest.Uwp', 'MSTest.WinUI') } { $ids = @('Microsoft.Testing.Extensions.TrxReport.Abstractions', 'Microsoft.Testing.Extensions.Telemetry', 'Microsoft.Testing.Platform.MSBuild', 'Microsoft.TestPlatform.ObjectModel') - if ($IsWindows) { $ids += 'MSTest.TestFramework' } + if ($IsWindows -or $Id -ne 'MSTest.TestAdapter') { $ids += 'MSTest.TestFramework' } Assert-Names $ids @($dependencies | ForEach-Object id) "$Id dependencies" - foreach ($dependency in $dependencies | Where-Object id -NE 'Microsoft.TestPlatform.ObjectModel') { + if ($platformFrameworks.ContainsKey($Id)) { + $core = @($dependencies | Where-Object id -CEQ 'MSTest.TestFramework') + if ($core.Count -ne 1 -or $core[0].version -cne "[$mstestVersion]") { + throw "$Id must pin the matching core package version exactly." + } + } + foreach ($dependency in $dependencies | Where-Object { $_.id -ne 'Microsoft.TestPlatform.ObjectModel' -and $_.id -notlike 'System.*' }) { if ($dependency.HasAttribute('exclude')) { throw "The adapter must retain build/analyzer assets from $($dependency.id)." } } } @@ -191,7 +241,7 @@ function Assert-PackageLayout($Archive, $Metadata, [string] $Id) { function New-Expectations([string] $Framework, [string] $Path) { $assemblies = [ordered]@{} - foreach ($id in $expectedIds | Where-Object { $_ -ne 'MSTest.Analyzers' }) { + foreach ($id in $portableIds | Where-Object { $_ -ne 'MSTest.Analyzers' }) { $archive = [System.IO.Compression.ZipFile]::OpenRead($packages[$id].Path) try { $names = switch ($id) { @@ -269,8 +319,8 @@ function Build-Consumer([string] $Name, [string] $Framework, [string] $PackageFe $assets = Get-Content -LiteralPath $assetsPath -Raw | ConvertFrom-Json -AsHashtable Assert-Names @([System.IO.Path]::TrimEndingDirectorySeparator($Cache)) @($assets.packageFolders.Keys | ForEach-Object { [System.IO.Path]::TrimEndingDirectorySeparator($_) }) "$Name restore folders" $resolved = @($assets.libraries.Keys | Where-Object { $_ -match '^(MSTest\.|Microsoft\.Testing\.)' }) - Assert-Names @($packages.Values | ForEach-Object { "$($_.Id)/$($_.Version)" }) $resolved "$Name native package closure" - foreach ($id in $expectedIds) { + Assert-Names @($portableIds | ForEach-Object { "$_/$($packages[$_].Version)" }) $resolved "$Name native package closure" + foreach ($id in $portableIds) { $package = $packages[$id] $cached = Join-Path $Cache ($id.ToLowerInvariant() + '\' + $package.Version + '\' + $id.ToLowerInvariant() + '.' + $package.Version + '.nupkg') $staged = Join-Path $PackageFeed "$id.$($package.Version).nupkg" @@ -289,7 +339,8 @@ function Run-Consumer($Consumer) { $expectations = Join-Path $Consumer.Directory 'expectations.json' New-Expectations $Consumer.Framework $expectations $env:TESTFX_BAZEL_MSTEST_EXPECTATIONS = $expectations - $env:TESTFX_BAZEL_CONSUMER_EVIDENCE = Join-Path $Consumer.Directory 'loaded-assemblies.json' + $env:TESTFX_BAZEL_CONSUMER_EVIDENCE = Join-Path $Consumer.Directory 'loaded-assemblies.paths' + $previousEvidence = if (Test-Path -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE) { @(Get-Content -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE).Count } else { 0 } $env:DOTNET_ROOT = Join-Path $directory 'missing-runtime' $env:DOTNET_ROOT_X64 = $env:DOTNET_ROOT $discovery = Invoke-Step "$name-discovery" $Consumer.Executable @('--list-tests') @@ -304,7 +355,10 @@ function Run-Consumer($Consumer) { '--report-trx', '--report-trx-filename', 'pass.trx', '--results-directory', $results ) Assert-Trx (Join-Path $results 'pass.trx') $passingTests - $provenance = Get-Content -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE -Raw | ConvertFrom-Json + $evidencePaths = @(Get-Content -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE) + if ($evidencePaths.Count -ne $previousEvidence + 1) { throw "$name did not append exactly one consumer provenance path." } + $provenancePath = $evidencePaths[-1] + $provenance = Get-Content -LiteralPath $provenancePath -Raw | ConvertFrom-Json $major = $Consumer.Framework.Substring(3).Split('.')[0] if ($provenance.Runtime -notlike ".NET $major.*" -or [System.IO.Path]::GetRelativePath($Consumer.Published, $provenance.CoreLib) -cne 'System.Private.CoreLib.dll') { throw "$name did not execute on its self-contained $($Consumer.Framework) runtime." @@ -316,7 +370,7 @@ function Run-Consumer($Consumer) { Assert-Trx (Join-Path $results 'failure.trx') @('IntentionalFailure') 1 'BAZEL_INTENTIONAL_FAILURE' $empty = Invoke-Step "$name-empty" $Consumer.Executable @('--filter', 'FullyQualifiedName~NoSuchPackageCase', '--minimum-expected-tests', '1') 9 if ($empty -notmatch 'tests ran 0, minimum expected 1') { throw "$name did not execute the zero-result policy." } - $runs.Add([pscustomobject]@{ Consumer = $Consumer; Runtime = $provenance.Runtime; Provenance = $env:TESTFX_BAZEL_CONSUMER_EVIDENCE; Results = $results; Passed = $true }) + $runs.Add([pscustomobject]@{ Consumer = $Consumer; Runtime = $provenance.Runtime; Provenance = $provenancePath; ProvenanceIndex = $env:TESTFX_BAZEL_CONSUMER_EVIDENCE; Results = $results; Passed = $true }) } function Run-MSBuildConsumer($Consumer) { @@ -328,6 +382,7 @@ function Run-MSBuildConsumer($Consumer) { @{ Name = 'failure'; Filter = 'FullyQualifiedName~IntentionalFailure'; Exit = 1; Tests = @('IntentionalFailure'); Failed = 1; Message = 'BAZEL_INTENTIONAL_FAILURE' } )) { $name = "msbuild-$($scenario.Name)" + $previousEvidence = if (Test-Path -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE) { @(Get-Content -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE).Count } else { 0 } $arguments = "--filter $($scenario.Filter) --report-trx --report-trx-filename $name.trx --results-directory `"$results`"" Push-Location $Consumer.Directory try { @@ -340,8 +395,11 @@ function Run-MSBuildConsumer($Consumer) { } finally { Pop-Location } Assert-Trx (Join-Path $results "$name.trx") $scenario.Tests $scenario.Failed $scenario.Message + $evidencePaths = @(Get-Content -LiteralPath $env:TESTFX_BAZEL_CONSUMER_EVIDENCE) + $expectedEvidence = $previousEvidence + $(if ($scenario.Name -eq 'pass') { 1 } else { 0 }) + if ($evidencePaths.Count -ne $expectedEvidence) { throw "$name did not preserve append-only consumer provenance." } } - $runs.Add([pscustomobject]@{ Consumer = $Consumer; Driver = 'Native Platform.MSBuild InvokeTestingPlatform'; ParentExits = @(0, 1); Results = $results; Passed = $true }) + $runs.Add([pscustomobject]@{ Consumer = $Consumer; Driver = 'Native Platform.MSBuild InvokeTestingPlatform'; ParentExits = @(0, 1); ProvenanceIndex = $env:TESTFX_BAZEL_CONSUMER_EVIDENCE; Provenance = $evidencePaths; Results = $results; Passed = $true }) } $environmentNames = @( @@ -365,11 +423,12 @@ try { [xml]$versions = Get-Content -LiteralPath (Join-Path $root 'eng\Versions.props') -Raw $mstestVersion = $versions.SelectSingleNode('//*[local-name()="VersionPrefix"]').InnerText + '-preview.bazel' $mtpVersion = $versions.SelectSingleNode('//*[local-name()="TestingPlatformVersionPrefix"]').InnerText + '-preview.bazel' - $null = Invoke-Step 'native-pack' $bazel @('build', '//:mstest_package_closure', '--jobs=2', '--noshow_progress', '--show_result=0') + $packageClosure = if ($IncludeWindowsPlatformPackages) { '//:mstest_platform_package_closure' } else { '//:mstest_package_closure' } + $null = Invoke-Step 'native-pack' $bazel @('build', $packageClosure, '--jobs=2', '--noshow_progress', '--show_result=0') $executionRoot = & $bazel info execution_root if ($LASTEXITCODE -ne 0) { throw 'Could not locate the native execution root.' } - $paths = @(& $bazel cquery //:mstest_package_closure --output=files --noshow_progress) - if ($LASTEXITCODE -ne 0 -or $paths.Count -ne $expectedIds.Count) { throw 'Expected exactly eight native packages in the MSTest validation closure.' } + $paths = @(& $bazel cquery $packageClosure --output=files --noshow_progress) + if ($LASTEXITCODE -ne 0 -or $paths.Count -ne $expectedIds.Count) { throw "Expected $($expectedIds.Count) native packages in the MSTest validation closure." } foreach ($path in $paths) { $packagePath = Join-Path $executionRoot $path $archive = [System.IO.Compression.ZipFile]::OpenRead($packagePath) @@ -402,7 +461,6 @@ try { finally { $archive.Dispose() } } Assert-Names $expectedIds @($packages.Keys) 'Native feed' - $sdkVersion = [regex]::Match((Get-Content -LiteralPath (Join-Path $root 'MODULE.bazel') -Raw), 'dotnet\.toolchain\(dotnet_version = "([^"]+)"\)').Groups[1].Value if (-not $sdkVersion) { throw 'Could not identify the native toolchain SDK.' } $dotnetName = if ($IsWindows) { 'dotnet.exe' } else { 'dotnet' } @@ -457,7 +515,7 @@ try { $expectations = Join-Path $counterfactual.Directory 'expectations.json' New-Expectations 'net11.0' $expectations $env:TESTFX_BAZEL_MSTEST_EXPECTATIONS = $expectations - $env:TESTFX_BAZEL_CONSUMER_EVIDENCE = Join-Path $counterfactual.Directory 'loaded-assemblies.json' + $env:TESTFX_BAZEL_CONSUMER_EVIDENCE = Join-Path $counterfactual.Directory 'loaded-assemblies.paths' $results = Join-Path $counterfactual.Directory 'results' $null = Invoke-Step 'counterfactual-execution' $counterfactual.Executable @( '--filter', 'FullyQualifiedName!~IntentionalFailure', '--minimum-expected-tests', '3', @@ -471,7 +529,7 @@ try { Run-Consumer $restored } $succeeded = $true - Write-Host "Native MSTest package consumers passed. Evidence: $directory" + Write-Host "Native MSTest ordinary package consumers passed. Optional UI packages cover modern native layouts, not classic UAP production or UI activation. Evidence: $directory" } finally { foreach ($name in $environmentNames) { @@ -480,7 +538,7 @@ finally { else { Remove-ProcessEnvironmentVariable $name } } Pop-Location - [pscustomobject]@{ Passed = $succeeded; BaseRevision = $revision; WorkingTree = $workingTree; Packages = @($packages.Values); Runs = $runs; Steps = $steps } | + [pscustomobject]@{ Passed = $succeeded; IncludeWindowsPlatformPackages = [bool]$IncludeWindowsPlatformPackages; ClassicUapProduced = $false; UIActivationValidated = $false; BaseRevision = $revision; WorkingTree = $workingTree; Packages = @($packages.Values); Runs = $runs; Steps = $steps } | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath (Join-Path $directory 'validation.json') -Encoding utf8 } diff --git a/eng/bazel/validate-sdk-packages.ps1 b/eng/bazel/validate-sdk-packages.ps1 index 2244d18042..608aa9eafd 100644 --- a/eng/bazel/validate-sdk-packages.ps1 +++ b/eng/bazel/validate-sdk-packages.ps1 @@ -131,7 +131,7 @@ function Assert-Trx([string] $Path, [string] $Name, [string] $Outcome) { Assert-Equal 0 ([int]$counters.notExecuted) 'Skipped test count' } -function New-Consumer([string] $Name, [string] $Properties, [switch] $Meta, [switch] $Layered) { +function New-Consumer([string] $Name, [string] $Properties, [switch] $Meta, [switch] $Layered, [string] $Framework = 'net11.0') { $path = Join-Path $directory $Name New-Item -ItemType Directory -Path $path | Out-Null foreach ($file in @('Directory.Build.props', 'Directory.Build.targets', 'Directory.Packages.props')) { @@ -143,7 +143,7 @@ function New-Consumer([string] $Name, [string] $Properties, [switch] $Meta, [swi 'msbuild-sdks' = @{ 'MSTest.Sdk' = $mstestVersion } test = @{ runner = $(if ($Meta) { 'VSTest' } else { 'Microsoft.Testing.Platform' }) } } | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath (Join-Path $path 'global.json') -Encoding utf8 - $propertiesXml = "net11.0enableenable$Properties" + $propertiesXml = "$Frameworkenableenable$Properties" $project = if ($Layered) { @" @@ -334,8 +334,19 @@ try { @{ Name = 'sdk-aot-selection'; Properties = 'true'; Expected = @('Microsoft.Testing.Platform.MSBuild', 'MSTest.TestFramework', 'MSTest.SourceGeneration', 'MSTest.TestAdapter', 'Microsoft.Testing.Extensions.TrxReport', 'Microsoft.Testing.Extensions.CodeCoverage', 'Microsoft.CodeCoverage.MSBuild') }, @{ Name = 'sdk-version-overrides'; Properties = '7.8.93.2.1'; MSTestVersion = '7.8.9'; PlatformVersion = '3.2.1'; Expected = @('MSTest.TestFramework', 'MSTest.TestAdapter', 'Microsoft.Testing.Extensions.TrxReport', 'Microsoft.Testing.Extensions.CodeCoverage') } ) + foreach ($framework in $frameworks) { + foreach ($model in @('Uwp', 'WinUI')) { + $evaluationCases += @{ + Name = "sdk-$($model.ToLowerInvariant())-$framework" + Framework = "$framework-windows10.0.26100.0" + Properties = "true$model" + Expected = @('MSTest.TestFramework', "MSTest.$model", 'Microsoft.Testing.Extensions.TrxReport', 'Microsoft.Testing.Extensions.CodeCoverage', 'Microsoft.Testing.Extensions.PackagedApp.MSBuild') + } + } + } foreach ($case in $evaluationCases) { - $consumer = New-Consumer $case.Name $case.Properties -Layered:([bool]$case.Layered) + $framework = if ($case.Framework) { $case.Framework } else { 'net11.0' } + $consumer = New-Consumer $case.Name $case.Properties -Layered:([bool]$case.Layered) -Framework $framework $result = Invoke-Step "evaluate-$($case.Name)" $dotnet @('msbuild', 'Consumer.csproj', '-nologo', '-getProperty:UsingMSTestSdk,MSTestVersion,MicrosoftTestingPlatformVersion,_MSTestSdkImportsMicrosoftNETSdk,OutputType,NETCoreSdkVersion,MSBuildToolsPath', '-getItem:PackageReference,PackageVersion', "-bl:$(Join-Path $directory "$($case.Name)-evaluation.binlog")") $consumer -Environment $consumerEnvironment diff --git a/eng/bazel/windows.bzl b/eng/bazel/windows.bzl index e3031f9739..59010b6c2e 100644 --- a/eng/bazel/windows.bzl +++ b/eng/bazel/windows.bzl @@ -6,7 +6,6 @@ load(":defs.bzl", "PORTABLE_FRAMEWORKS", "testfx_library", "testfx_matrix") _PLATFORM = "src/Platform/Microsoft.Testing.Platform" _SHARED = "src/Platform/SharedExtensionHelpers" _PACKAGED_APP = "src/Platform/Microsoft.Testing.Extensions.PackagedApp" -_FRAMEWORK = "src/TestFramework/TestFramework" _UI_AUTOMATION = "src/TestFramework/TestFramework.Windows.UIAutomation" _PACKAGED_FRAMEWORKS = [tfm + "-windows10.0.19041.0" for tfm in PORTABLE_FRAMEWORKS] _DESKTOP_FRAMEWORKS = [tfm + "-windows7.0" for tfm in PORTABLE_FRAMEWORKS] @@ -104,53 +103,17 @@ def _windows_library(name, target_frameworks, linked_srcs = [], minimum_version ) def _mstest_flavor(flavor, framework, symbol, projection_deps): - core = "framework_" + flavor - compatibility = "framework_compatibility_" + flavor - extensions = "framework_extensions_" + flavor + attributes = "mstest_" + flavor + assembly_name = "MSTest.Uwp" if flavor == "uwp" else "MSTest.WinUI" services = "platform_services_" + flavor adapter = "adapter_" + flavor - ui_assembly = "MSTest.Uwp" if flavor == "uwp" else "MSTest.WinUI" - framework_project = "src/TestFramework/" + ui_assembly + framework_project = "src/TestFramework/" + assembly_name services_project = "src/Adapter/MSTestAdapter.PlatformServices" adapter_project = "src/Adapter/MSTest.TestAdapter" _windows_library( - name = core, - project = _FRAMEWORK, - assembly_name = "MSTest.TestFramework", - target_frameworks = [framework], - minimum_version = "10.0.17763.0" if flavor == "winui" else "", - defines = ["EXCLUDE_RANGE_INDEX_POLYFILL", "EXCLUDE_OS_POLYFILL", "TESTFRAMEWORK_CI_DETECTOR", symbol], - linked_srcs = [ - _PLATFORM + "/Helpers/RoslynHashCode.cs", - _PLATFORM + "/Helpers/CIEnvironmentDetector.cs", - ], - resources = {_FRAMEWORK + "/Resources/FrameworkMessages.resx": "Microsoft.VisualStudio.TestTools.UnitTesting.Resources.FrameworkMessages"}, - usings = ["Polyfills", "Microsoft.VisualStudio.TestTools.UnitTesting.Resources"], - internals_visible_to = [ - "DynamicProxyGenAssembly2", - "MSTestAdapter.PlatformServices", - "MSTest.TestAdapter", - "Microsoft.VisualStudio.TestPlatform.MSTestAdapter.UnitTests", - "MSTest.Extensions.Hosting", - "Microsoft.VisualStudio.TestPlatform.TestFramework.UnitTests", - "MSTestAdapter.PlatformServices.UnitTests", - "MSTest.Performance.Benchmarks", - ], - deps = [":assert_generator_plugin"], - ) - _windows_library( - name = compatibility, - project = "src/TestFramework/TestFramework.Compatibility", - assembly_name = "MSTest.TestFramework.Extensions", - target_frameworks = [framework], - emit_supported_os = False, - polyfills = False, - deps = [":" + core], - ) - _windows_library( - name = extensions, + name = attributes, project = framework_project, - assembly_name = ui_assembly, + assembly_name = assembly_name, target_frameworks = [framework], # The WinUI source already carries its supported-OS annotation. emit_supported_os = flavor != "winui", @@ -158,16 +121,7 @@ def _mstest_flavor(flavor, framework, symbol, projection_deps): usings = ["Polyfills", "Microsoft.VisualStudio.TestTools.UnitTesting.Resources"], resources = {framework_project + "/Resources/FrameworkExtensionsMessages.resx": "Microsoft.VisualStudio.TestTools.UnitTesting.Resources.FrameworkExtensionsMessages"} if flavor == "winui" else {}, analyzer_configs = ["//eng/bazel/windows:winrt.globalconfig"], - embedded_resources = { - framework_project + "/Properties/MSTest.Uwp.rd.xml": "Microsoft.VisualStudio.TestTools.UnitTesting.Properties.MSTest.Uwp.rd.xml", - } if flavor == "uwp" else {}, - internals_visible_to = [ - "MSTestAdapter.PlatformServices", - "MSTestAdapter.PlatformServices.UnitTests", - "MSTest.TestAdapter", - "Microsoft.VisualStudio.TestPlatform.MSTestAdapter.UnitTests", - ], - deps = [":" + core] + projection_deps, + deps = [":framework"] + projection_deps, ) _windows_library( name = services, @@ -188,7 +142,7 @@ def _mstest_flavor(flavor, framework, symbol, projection_deps): "MSTest.Acceptance.IntegrationTests", "MSTest.Performance.Benchmarks", ], - deps = [":" + core, ":" + extensions] + projection_deps, + deps = [":framework"] + projection_deps, ) _windows_library( name = adapter, @@ -220,8 +174,7 @@ def _mstest_flavor(flavor, framework, symbol, projection_deps): deps = [ ":platform", ":" + services, - ":" + core, - ":" + extensions, + ":framework", ":trx_abstractions", ":trx", ":telemetry", @@ -231,7 +184,7 @@ def _mstest_flavor(flavor, framework, symbol, projection_deps): "@paket.source//microsoft.testplatform.objectmodel", ], ) - native.filegroup(name = "_windows_" + flavor, srcs = [":" + core, ":" + extensions, ":" + services, ":" + adapter], tags = ["manual"]) + native.filegroup(name = "_windows_" + flavor, srcs = [":framework", ":" + attributes, ":" + services, ":" + adapter], tags = ["manual"]) testfx_matrix(name = "windows_" + flavor, target = ":_windows_" + flavor, frameworks = [framework]) def _library_probe(name, framework, scenario, deps, project_sdk = "default"): @@ -434,13 +387,13 @@ def windows_libraries(): "windows_probe_uwp", "net10.0-windows10.0.17763.0", "uwp", - [":adapter_uwp", ":platform_services_uwp", ":framework_extensions_uwp", ":framework_uwp", ":framework_compatibility_uwp"], + [":adapter_uwp", ":platform_services_uwp", ":mstest_uwp", ":framework", ":framework_compatibility"], ) _library_probe( "windows_probe_winui", "net10.0-windows10.0.18362.0", "winui", - [":adapter_winui", ":platform_services_winui", ":framework_extensions_winui", ":framework_winui", ":framework_compatibility_winui"], + [":adapter_winui", ":platform_services_winui", ":mstest_winui", ":framework", ":framework_compatibility"], ) native.filegroup( name = "windows_probes", diff --git a/eng/bazel/windows/AppModelHost.cs b/eng/bazel/windows/AppModelHost.cs index 2fc2bd1e7f..cbc5e95a17 100644 --- a/eng/bazel/windows/AppModelHost.cs +++ b/eng/bazel/windows/AppModelHost.cs @@ -57,7 +57,14 @@ private static void WriteProvenance() }), }; string json = JsonSerializer.Serialize(report); - File.WriteAllText(Path.Combine(AppContext.BaseDirectory, "native-app-model.json"), json); + string reportPath = Path.Join(AppContext.BaseDirectory, "native-app-model-" + Guid.NewGuid().ToString("N") + ".json"); + using (FileStream stream = File.Open(reportPath, FileMode.CreateNew, FileAccess.Write, FileShare.Read)) + using (var writer = new StreamWriter(stream)) + { + writer.Write(json); + } + + File.AppendAllLines(Path.Join(AppContext.BaseDirectory, "native-app-model.paths"), [reportPath]); Console.WriteLine("BAZEL_APPMODEL_PROVENANCE: " + json); } diff --git a/eng/bazel/windows/LibraryProbe.cs b/eng/bazel/windows/LibraryProbe.cs index d658c34736..8af96a0ead 100644 --- a/eng/bazel/windows/LibraryProbe.cs +++ b/eng/bazel/windows/LibraryProbe.cs @@ -65,16 +65,29 @@ public static void Main() #else scenario = "winui"; #endif - Assembly extensions = Assembly.Load(scenario == "uwp" ? "MSTest.Uwp" : "MSTest.WinUI"); - Assembly framework = VerifyFrameworkApi(extensions); - Type uiMethod = extensions.GetType( + string uiAssemblyName = scenario == "uwp" ? "MSTest.Uwp" : "MSTest.WinUI"; + Assembly attributes = Assembly.Load(uiAssemblyName); + Assembly framework = VerifyFrameworkApi(attributes); + Type uiMethod = attributes.GetType( "Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute", throwOnError: true)!; - if (!typeof(Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute).IsAssignableFrom(uiMethod)) + if (uiMethod.Assembly != attributes || !typeof(Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute).IsAssignableFrom(uiMethod)) { - throw new InvalidOperationException("The Windows flavor did not expose the UI test method attribute."); + throw new InvalidOperationException($"The UI test method attribute must be defined in {uiAssemblyName} and derive from shared core."); } - sourceAssemblies.Add(extensions); + const string winuiTargetName = "Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.WinUITestTargetAttribute"; + if ((attributes.GetType(winuiTargetName) is not null) != (scenario == "winui")) + { + throw new InvalidOperationException("WinUITestTargetAttribute must be isolated to MSTest.WinUI."); + } + + string otherFlavor = scenario == "uwp" ? "MSTest.WinUI" : "MSTest.Uwp"; + if (File.Exists(Path.Join(AppContext.BaseDirectory, otherFlavor + ".dll"))) + { + throw new InvalidOperationException("The Windows consumer contains the other flavor's UI attributes."); + } + + sourceAssemblies.Add(attributes); sourceAssemblies.Add(framework); sourceAssemblies.Add(Assembly.Load("MSTestAdapter.PlatformServices")); sourceAssemblies.Add(Assembly.Load("MSTest.TestAdapter")); @@ -84,6 +97,7 @@ public static void Main() #if WINDOWS_DESKTOP_PROBE || WINDOWS_UWP_PROBE || WINDOWS_WINUI_PROBE sourceAssemblies.Add(Assembly.Load("MSTest.TestFramework.Extensions")); #endif + foreach (Assembly assembly in sourceAssemblies) { if (!assembly.GetCustomAttributes() @@ -104,27 +118,34 @@ public static void Main() } #if WINDOWS_DESKTOP_PROBE || WINDOWS_UWP_PROBE || WINDOWS_WINUI_PROBE - private static Assembly VerifyFrameworkApi(Assembly? extensions = null) + private static Assembly VerifyFrameworkApi(Assembly? attributes = null) { Assembly framework = typeof(Microsoft.VisualStudio.TestTools.UnitTesting.Assert).Assembly; Assembly compatibility = Assembly.Load("MSTest.TestFramework.Extensions"); Type context = typeof(Microsoft.VisualStudio.TestTools.UnitTesting.TestContext); if (framework.GetName().Name != "MSTest.TestFramework" || context.Assembly != framework || compatibility.GetType(context.FullName!) != context - || extensions?.GetType(context.FullName!) is not null) + || attributes?.GetType(context.FullName!) is not null + || framework.GetCustomAttribute() is not null) { - throw new InvalidOperationException("TestContext must be defined in core and forwarded only by the compatibility facade."); + throw new InvalidOperationException("TestContext must be defined in the ordinary shared core and forwarded by the compatibility facade."); } const string deploymentItemName = "Microsoft.VisualStudio.TestTools.UnitTesting.DeploymentItemAttribute"; Type? deploymentItem = framework.GetType(deploymentItemName); if (deploymentItem is null || deploymentItem.Assembly != framework || compatibility.GetType(deploymentItemName) != deploymentItem - || extensions?.GetType(deploymentItemName) is not null) + || attributes?.GetType(deploymentItemName) is not null) { throw new InvalidOperationException("DeploymentItemAttribute must be defined in shared modern core and forwarded by the compatibility facade."); } + if (compatibility.DefinedTypes.Any(type => type.IsPublic || type.IsNestedPublic) + || compatibility.GetForwardedTypes().Any(type => type.Assembly != framework)) + { + throw new InvalidOperationException("The compatibility facade must only forward non-UI types to the ordinary shared core."); + } + string[] directories = ["TestRunDirectory", "DeploymentDirectory", "ResultsDirectory", "TestRunResultsDirectory", "TestResultsDirectory", "TestTempDirectory"]; foreach (string directory in directories) { @@ -134,6 +155,12 @@ private static Assembly VerifyFrameworkApi(Assembly? extensions = null) } } + if (framework.GetType("Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute") is not null + || framework.GetType("Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.WinUITestTargetAttribute") is not null) + { + throw new InvalidOperationException("UI-specific attributes must not be defined in shared core."); + } + return framework; } #endif diff --git a/eng/bazel/windows/targeting_tests.bzl b/eng/bazel/windows/targeting_tests.bzl index 162f0966b0..454562c13d 100644 --- a/eng/bazel/windows/targeting_tests.bzl +++ b/eng/bazel/windows/targeting_tests.bzl @@ -17,6 +17,7 @@ load( ) load("@rules_dotnet//dotnet/private:providers.bzl", "DotnetAssemblyCompileInfo", "DotnetAssemblyRuntimeInfo") load("@rules_dotnet//dotnet/private/transitions:common.bzl", "FRAMEWORK_COMPATABILITY_TRANSITION_OUTPUTS", "rid_compatability_transition_outputs") +load("//eng/bazel:mstest-packages.bzl", "mstest_package_tfm") def _configuration(framework): # Windows probes have no host apphost/runtime packs on Linux or macOS, so pin @@ -37,26 +38,26 @@ def _flavor_targets(flavor, framework): providers = [DotnetAssemblyCompileInfo, DotnetAssemblyRuntimeInfo], ) for name, target in [ - ("extensions", "framework_extensions_" + flavor), + ("attributes", "mstest_" + flavor), ("services", "platform_services_" + flavor), ("adapter", "adapter_" + flavor), ("probe", "windows_probe_" + flavor + "_app"), ] } -def _core_owners(target): +def _assembly_owners(target, name): compile_info = target[DotnetAssemblyCompileInfo] runtime_info = target[DotnetAssemblyRuntimeInfo] compile_owners = depset([ str(reference.owner) for reference in compile_info.refs + compile_info.transitive_refs.to_list() - if reference.basename == "MSTest.TestFramework.dll" + if reference.basename == name + ".dll" ]) runtime_owners = depset([ str(library.owner) for dependency in [runtime_info] + runtime_info.deps.to_list() for library in dependency.libs - if library.basename == "MSTest.TestFramework.dll" + if library.basename == name + ".dll" ]) return sorted(compile_owners.to_list()), sorted(runtime_owners.to_list()) @@ -91,19 +92,47 @@ def _targeting_tests_impl(ctx): {"name": "Microsoft.NETCore.App", "version": "8.0.0"}, {"name": "Microsoft.WindowsDesktop.App", "version": "8.0.0"}, ], desktop["runtimeOptions"]["frameworks"]), + ("Net11UwpConsumesNet10Attributes", "net10.0-windows10.0.17763.0", get_nearest_compatible_target_framework("net11.0-windows10.0.17763.0", ["net10.0-windows10.0.17763.0"])), + ("Net11WinuiConsumesNet10Attributes", "net10.0-windows10.0.18362.0", get_nearest_compatible_target_framework("net11.0-windows10.0.18362.0", ["net10.0-windows10.0.18362.0"])), + ("ModernUiSelectsOrdinaryCore", "net11.0", get_nearest_compatible_target_framework("net11.0-windows10.0.18362.0", ["netstandard2.0", "net462", "net10.0", "net11.0"])), + ("UwpLibPathUsesNuGetCanonicalWindowsVersion", "net10.0-windows10.0.17763", mstest_package_tfm("net10.0-windows10.0.17763.0")), + ("WinuiLibPathUsesNuGetCanonicalWindowsVersion", "net10.0-windows10.0.18362", mstest_package_tfm("net10.0-windows10.0.18362.0")), + ("ClassicUwpLibPathPreservesMinimum", "uap10.0.16299", mstest_package_tfm("uap10.0.16299")), ] - for flavor, core, targets in [ - ("Desktop", "framework", [ctx.attr.desktop[0]]), - ("Uwp", "framework_uwp", [ctx.attr.uwp_extensions[0], ctx.attr.uwp_services[0], ctx.attr.uwp_adapter[0], ctx.attr.uwp_probe[0]]), - ("Winui", "framework_winui", [ctx.attr.winui_extensions[0], ctx.attr.winui_services[0], ctx.attr.winui_adapter[0], ctx.attr.winui_probe[0]]), + for flavor, targets in [ + ("Desktop", [ctx.attr.desktop[0]]), + ("Uwp", [ctx.attr.uwp_attributes[0], ctx.attr.uwp_services[0], ctx.attr.uwp_adapter[0], ctx.attr.uwp_probe[0]]), + ("Winui", [ctx.attr.winui_attributes[0], ctx.attr.winui_services[0], ctx.attr.winui_adapter[0], ctx.attr.winui_probe[0]]), ]: for target in targets: - compile_owners, runtime_owners = _core_owners(target) - expected = [str(Label("//:" + core))] + compile_owners, runtime_owners = _assembly_owners(target, "MSTest.TestFramework") + expected = [str(Label("//:framework"))] cases.extend([ ("{} {} compile core".format(flavor, target.label), expected, compile_owners), ("{} {} runtime core".format(flavor, target.label), expected, runtime_owners), ]) + compile_owners, runtime_owners = _assembly_owners(ctx.attr.desktop[0], "MSTest.TestFramework.Extensions") + expected = [str(Label("//:framework_compatibility"))] + cases.extend([ + ("Desktop compile compatibility facade", expected, compile_owners), + ("Desktop runtime compatibility facade", expected, runtime_owners), + ]) + for flavor, assembly_name, other in [("uwp", "MSTest.Uwp", "MSTest.WinUI"), ("winui", "MSTest.WinUI", "MSTest.Uwp")]: + for kind in ["attributes", "services", "adapter", "probe"]: + target = getattr(ctx.attr, flavor + "_" + kind)[0] + for name, owner, present in [ + (assembly_name, "mstest_" + flavor, kind in ["attributes", "probe"]), + (other, "", False), + ("MSTest.TestAdapter", "adapter_" + flavor, kind in ["adapter", "probe"]), + ("MSTestAdapter.PlatformServices", "platform_services_" + flavor, kind in ["services", "adapter", "probe"]), + ("MSTest.TestFramework.Extensions", "framework_compatibility", kind == "probe"), + ]: + compile_owners, runtime_owners = _assembly_owners(target, name) + expected = [str(Label("//:" + owner))] if present else [] + cases.extend([ + ("{} {} compile {}".format(flavor, kind, name), expected, compile_owners), + ("{} {} runtime {}".format(flavor, kind, name), expected, runtime_owners), + ]) failures = [ "{}: expected {}, actual {}".format(name, repr(expected), repr(actual)) for name, expected, actual in cases diff --git a/eng/bazel/windows/validate-app-model.ps1 b/eng/bazel/windows/validate-app-model.ps1 index 7eb3a467ef..9da26539c3 100644 --- a/eng/bazel/windows/validate-app-model.ps1 +++ b/eng/bazel/windows/validate-app-model.ps1 @@ -24,6 +24,7 @@ $directory = Join-Path $ArtifactsDirectory $run $layout = Join-Path $directory 'package layout' $identity = "TestFx.Bazel.$run" $steps = [Collections.Generic.List[object]]::new() +$recordedProvenancePaths = @() $succeeded = $false $inputHashes = @(Get-FileHash -Algorithm SHA256 -LiteralPath @( (Join-Path $root 'MODULE.bazel'), (Join-Path $root 'eng\bazel\defs.bzl'), @@ -63,6 +64,7 @@ function Invoke-Process([string] $Name, [string] $Executable, [string[]] $Argume if ($timedOut -or $process.ExitCode -ne $ExpectedExit) { throw "$Name returned $($process.ExitCode), expected $ExpectedExit; timeout=$timedOut. See its retained log." } + if ($text.Contains('BAZEL_APPMODEL_PROVENANCE: ')) { $null = Read-AppModelProvenance -RequireNew } if ($StandardOutputOnly) { return $output } return $text } @@ -108,6 +110,21 @@ function Assert-Pass([string] $Path) { } } +function Read-AppModelProvenance([switch] $RequireNew) { + $paths = @(Get-Content -LiteralPath (Join-Path $layout 'native-app-model.paths')) + if ($paths.Count -eq 0 -or $paths.Count -lt $script:recordedProvenancePaths.Count -or + ($RequireNew -and $paths.Count -le $script:recordedProvenancePaths.Count)) { + throw 'The app-model host did not append a provenance path.' + } + for ($index = 0; $index -lt $script:recordedProvenancePaths.Count; $index++) { + if ($paths[$index] -cne $script:recordedProvenancePaths[$index]) { + throw 'The app-model host overwrote an earlier provenance path.' + } + } + $script:recordedProvenancePaths = $paths + return Get-Content -LiteralPath $paths[-1] -Raw | ConvertFrom-Json +} + Push-Location $root try { $targets = @('//:windows_app_model') @@ -174,7 +191,7 @@ try { $activated = Join-Path $directory 'activated' $null = Invoke-Process 'registered-activation' $hostLayout.Executable ($pass + @('--results-directory', $activated)) Assert-Pass (Join-Path $activated 'tests.trx') - $marker = Get-Content -LiteralPath (Join-Path $layout 'native-app-model.json') -Raw | ConvertFrom-Json + $marker = Read-AppModelProvenance $packages = Get-OwnedPackage if ($packages.Count -ne 1 -or $marker.PackageName -cne $identity -or -not [string]::Equals($packages[0].InstallLocation.TrimEnd('\'), $layout.TrimEnd('\'), [StringComparison]::OrdinalIgnoreCase) -or @@ -207,9 +224,9 @@ try { } finally { try { - $markerPath = Join-Path $layout 'native-app-model.json' + $markerPath = Join-Path $layout 'native-app-model.paths' if (Test-Path -LiteralPath $markerPath) { - $marker = Get-Content -LiteralPath $markerPath -Raw | ConvertFrom-Json + $marker = Read-AppModelProvenance $process = Get-CimInstance Win32_Process -Filter "ProcessId=$($marker.ProcessId)" if ($process -and [string]::Equals($process.ExecutablePath, (Join-Path $layout 'TestFx.Bazel.AppModelHost.exe'), [StringComparison]::OrdinalIgnoreCase)) { Stop-Process -Id $marker.ProcessId diff --git a/eng/bazel/windows/validate.ps1 b/eng/bazel/windows/validate.ps1 index 972dae3d14..627c1bedac 100644 --- a/eng/bazel/windows/validate.ps1 +++ b/eng/bazel/windows/validate.ps1 @@ -22,6 +22,7 @@ $directory = Join-Path $ArtifactsDirectory $run New-Item -ItemType Directory -Path $directory | Out-Null $steps = [Collections.Generic.List[object]]::new() $consumers = [Collections.Generic.List[object]]::new() +$coreHashes = @{} $succeeded = $false $inputHashes = @(Get-FileHash -Algorithm SHA256 -LiteralPath @( (Join-Path $root 'MODULE.bazel'), (Join-Path $root 'BUILD.bazel'), @@ -223,15 +224,27 @@ try { } $windowsAssemblies = @($report.SourceAssemblies | Where-Object { $_.Name -in @('Microsoft.Testing.Extensions.PackagedApp', 'MSTest.Windows.UIAutomation') -or - ($probe.Scenario -in @('uwp', 'winui') -and $_.Name -in @('MSTest.TestFramework', 'MSTest.Uwp', 'MSTest.WinUI', 'MSTestAdapter.PlatformServices', 'MSTest.TestAdapter')) + ($probe.Scenario -in @('uwp', 'winui') -and $_.Name -in @('MSTest.Uwp', 'MSTest.WinUI', 'MSTestAdapter.PlatformServices', 'MSTest.TestAdapter')) }) foreach ($assembly in $windowsAssemblies) { - $minimum = if ($probe.Scenario -eq 'winui' -and $assembly.Name -in @('MSTest.TestFramework', 'MSTest.WinUI')) { '10.0.17763.0' } else { $probe.Platform } + $minimum = if ($assembly.Name -eq 'MSTest.WinUI') { '10.0.17763.0' } else { $probe.Platform } if ($assembly.Framework -cne ".NETCoreApp,Version=v$($probe.Major).0" -or $assembly.Platform -cne "Windows$($probe.Platform)" -or $assembly.SupportedOS -cne "windows$minimum") { throw "Windows TFM/minimum metadata mismatch in $($assembly.Name)." } } + foreach ($core in @($report.SourceAssemblies | Where-Object { $_.Name -in @('MSTest.TestFramework', 'MSTest.TestFramework.Extensions') })) { + $expectedCoreFramework = ".NETCoreApp,Version=v$($probe.Major).0" + # Portable native shared assemblies can omit this attribute; Windows metadata is never valid here. + if (($core.Framework -and $core.Framework -cne $expectedCoreFramework) -or $core.Platform -or $core.SupportedOS) { + throw "A UI-specific $($core.Name) was selected instead of the ordinary shared assembly." + } + $key = "$($core.Name)|$expectedCoreFramework" + if ($coreHashes.ContainsKey($key) -and $coreHashes[$key] -cne $core.Sha256) { + throw "Desktop and UI consumers did not load the same native $($core.Name) payload." + } + $coreHashes[$key] = $core.Sha256 + } if ($probe.Scenario -eq 'packaged' -and (Test-Path -LiteralPath (Join-Path $layout 'Microsoft.Windows.UI.Xaml.dll'))) { throw 'The ordinary WinRT profile leaked the UWP XAML projection.' } @@ -243,7 +256,7 @@ try { if ($LASTEXITCODE -ne 0) { throw 'Could not inspect the Windows producer action graph.' } $graph = Get-Content -LiteralPath $graphPath -Raw | ConvertFrom-Json $compilerCount = @($graph.actions | Where-Object mnemonic -eq 'CSharpCompile').Count - if ($compilerCount -lt 16) { throw 'The Windows library matrix is missing compiler actions.' } + if ($compilerCount -eq 0) { throw 'The Windows library matrix has no native compiler actions.' } foreach ($action in $graph.actions) { foreach ($argument in $action.arguments) { if ($argument -match '(?i)(^|[\s/\\])(MSBuild(\.exe|\.dll)?|dotnet(\.exe)?\s+(build|test|msbuild))(\s|$)') { diff --git a/test/UnitTests/Bazel.SdkPackages.UnitTests/SdkPackageTests.cs b/test/UnitTests/Bazel.SdkPackages.UnitTests/SdkPackageTests.cs index 39b8ff88f0..24fd40c317 100644 --- a/test/UnitTests/Bazel.SdkPackages.UnitTests/SdkPackageTests.cs +++ b/test/UnitTests/Bazel.SdkPackages.UnitTests/SdkPackageTests.cs @@ -105,6 +105,9 @@ public async Task Generate_MetaPackagePreservesDependencyAndAssetFlow() [DataRow("framework-drift", "differs from SupportedNetFrameworks")] [DataRow("meta-private-assets", "test SDK build-asset flow changed")] [DataRow("meta-reference", "Unsupported meta-package project reference")] + [DataRow("meta-retired-framework", "Unsupported meta-package project reference")] + [DataRow("meta-ui-reference", "Unsupported meta-package project reference")] + [DataRow("meta-ui-condition", "Unsupported meta-package project reference")] public async Task Generate_RejectsContractDriftBeforeWritingOutputs(string scenario, string expectedError) { JsonObject input = CreateInput(sdk: !scenario.StartsWith("meta-", StringComparison.Ordinal)); @@ -140,6 +143,15 @@ public async Task Generate_RejectsContractDriftBeforeWritingOutputs(string scena case "meta-reference": ReplaceInFile(PathFor("Package.csproj"), "MSTest.TestAdapter.csproj", "Unknown.csproj"); break; + case "meta-retired-framework": + ReplaceInFile(PathFor("Package.csproj"), @"TestFramework\TestFramework\TestFramework.csproj", @"TestFramework\TestFramework.Extensions\TestFramework.Extensions.csproj"); + break; + case "meta-ui-reference": + ReplaceInFile(PathFor("Package.csproj"), @"MSTest.Uwp\MSTest.Uwp.csproj", @"MSTest.WinUI\MSTest.WinUI.csproj"); + break; + case "meta-ui-condition": + ReplaceInFile(PathFor("Package.csproj"), "'$(TargetFramework)' == '$(UwpMinimum)'", "'$(TargetFramework)' != '$(UwpMinimum)'"); + break; default: Assert.Fail($"Unknown scenario: {scenario}"); break;