Skip to content

Use a fallback class name in TRX reports for nodes without type info - #12466

Open
Amaury Levé (Evangelink) wants to merge 3 commits into
mainfrom
dev/amauryleve/trx-fallback-class-name
Open

Amaury Levé (Evangelink) wants to merge 3 commits into
mainfrom
dev/amauryleve/trx-fallback-class-name

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Pull request

Summary

Related to #12436 ("Which properties are required for Microsoft.Testing.Extensions.TrxReport?"). This PR does not close it; see "Out of scope" below.

Problem. A TestNode that has a final state but neither TrxFullyQualifiedTypeNameProperty nor TestMethodIdentifierProperty makes TrxReportEngine.GetClassAndMethodName throw InvalidOperationException ("The test framework '...' with UID '...' does not support the 'ITrxReportCapability' leading to missing or incomplete TRX reports"). That is the shape reported by frameworks without source or type information (the issue notes xUnit.net supports test cases that don't come from source code at all). With the released 2.5.3 package the exception escapes report generation: the test host process crashes, the test application exits with code 7 and the TRX contains no test results (reproduced below).

Change. When neither property is present, the class name is now the documented placeholder UnknownClass, and the method name is the node's display name (already the behavior of the caller, testMethodName ?? testResultDisplayName).

  • TrxReportEngine.cs / TrxReportEngine.Results.cs: new private const string FallbackClassName = "UnknownClass". GetClassAndMethodName returns (FallbackClassName, null) instead of throwing when there is no method identifier (and no TrxFullyQualifiedTypeNameProperty). The method no longer uses instance state, so it became static, and the now-unused resources using was removed. The TrxReportFrameworkDoesNotSupportTrxReportCapability resource is still used by TrxDataConsumer for the "capability not supported" warning, so there are no .resx/.xlf changes.
  • Unchanged: nodes that carry either property (TrxFullyQualifiedTypeNameProperty still wins, then namespace + type name from the method identifier); the exceptions for a missing final state and for duplicate singleton properties.
  • Docs: the ITrxReportCapability XML documentation and the Microsoft.Testing.Extensions.TrxReport.Abstractions package readme (PACKAGE.md) now say which properties are required (the final state), recommended (TrxFullyQualifiedTypeNameProperty or TestMethodIdentifierProperty, with this fallback), optional (everything else), and that duplicates of consumed singleton properties make report generation fail, while unconsumed properties are ignored. This extends the documentation added in Document required and optional TRX report metadata #12458 and removes its now-stale sentence "Omitting both identity properties causes report generation to fail."
  • Tests: 4 new test cases in the existing TRX unit tests (see "Test-value evidence").
  • History: the branch was rebased onto main at ddb33a558d5200030053e571dba7a65de476f317. The original fix is now 9d8106dabba1abd0e60b2dc7726232734a87fa24. After review, d7e51a5391efbcb89d14142f2654914fea1e8418 renamed the placeholder from DefaultClassName to UnknownClass and removed the VSTest-parity claim from the docs. The demonstration and validation below were re-run on d7e51a5391efbcb89d14142f2654914fea1e8418.
  • No public API change: PublicAPI.*.txt, resources and global.json are untouched. No changelog entry: docs/Changelog-Platform.md is updated in bulk at release time and recent TRX fix PRs don't touch it.

Out of scope (why this is "Related to", not "Fixes"). The issue discussion also describes (1) a second crash: repeated results with the same UID that mix an explicit TrxTestDefinitionName with an omitted one whose display-name fallback differs, and (2) a proposal to report genuine contract violations as controlled reporting failures (preserve recoverable results, mark the run unsuccessful, record the affected IDs in RunInfo, nonzero exit code). This PR implements the first part of the stated preference ("support source-less tests through a documented fallback") only.

Design and decision record

This is a bug fix, not a new feature, so there is no separate design document; the rationale is here.

  • Why a fallback instead of keeping the throw. TestMethod/@className is use="required" in the TRX schema (vstst.xsd, vendored copy; the XSD ships with Visual Studio and is not in this repo), so the report needs some class name, but a test node is not obliged to have one. The engine already accepts a missing method identifier whenever TrxFullyQualifiedTypeNameProperty is present (the method name falls back to the display name, which is how NUnit is reported today), so "neither property" was the only remaining hard failure. Failing the whole report, with exit code 7 and an empty TRX, for a node that has a valid final state is disproportionate.
  • A comment I changed. The NOTE above that display-name fallback ended with "However, this is very wrong and we probably should fail if TestMethodIdentifierProperty isn't present." This PR deliberately does the opposite for source-less nodes (the method-name fallback itself is unchanged), so I replaced that sentence with the new decision instead of leaving a comment that contradicts the code. Easy to restore if you would rather keep the original remark.
  • Why UnknownClass. The name says what happened: the framework reported no type information, so the class is unknown. It does not read like a real default or a real type. It is a single identifier token, so consumers that split className on . into namespace and type still get a valid, unambiguous value. There was no existing constant or convention in src/Platform to reuse.
  • Why not DefaultClassName (this PR's first revision). It is the initial value in the VSTest TRX logger's Converter.GetTestClassName (microsoft/vstest@0f5ac692, Converter.cs#L593-L634). VSTest only returns it when the fully qualified name has no . or :: and the test source file name is also unusable. The parity argument was therefore weak, and the name hides the fact that the type information is missing.
  • Why not an "error" name (for example ErrorUnknownClass). Source-less tests are legitimate; xUnit.net supports test cases that don't come from source code. TRX also has a real Error outcome, so an error-sounding class name would suggest a failure that did not happen. Nothing is wrong with the result itself, only the grouping information is absent.
  • Alternatives considered (not implemented).
    • Module/assembly name as the class name (VSTest's choice before its DefaultClassName fallback): environment-dependent, adds edge cases (empty or odd names), and looks like a real type.
    • Node UID or display name as the class name: creates one TRX class per test and UID formats are framework-specific. Frameworks that want a specific grouping can still supply TrxFullyQualifiedTypeNameProperty, which keeps precedence (the issue author says they will do that framework-side).
    • An empty class name: the attribute is required and an empty value is not useful to TRX consumers.
    • Keeping the throw with a clearer message: still crashes the host for a valid source-less node.
  • Known trade-offs. Multiple class-less nodes in one test app share the UnknownClass class; each is still its own test definition with its own test id. A real type named UnknownClass in the global namespace would share that class name; that is unlikely, and its tests would still have distinct test definitions. An empty fully qualified type name, or a method identifier with an empty type name, can still produce an empty class name (pre-existing and unchanged).

Demonstration

Real composed product, no mocks: a minimal Microsoft Testing Platform host (source at the end of this section) whose test framework reports only a final state and timing, the shape from the issue, plus the real Microsoft.Testing.Extensions.TrxReport assemblies. The same host source was built twice in a scratch project outside the repo (nothing of it is committed): once against the released package, once against this branch's build output.

Before: released Microsoft.Testing.Extensions.TrxReport 2.5.3 from nuget.org (the shipped bits users have, not this PR's merge base):

$env:E2E_SHAPE = 'state-only'
.\E2EReleased.exe --report-trx --report-trx-filename e2e.trx --results-directory results-before
Unhandled exception. System.InvalidOperationException: The test framework 'StateOnlyTestFramework' with UID 'StateOnlyTestFramework' does not support the 'ITrxReportCapability' leading to missing or incomplete TRX reports
   at Microsoft.Testing.Extensions.TrxReport.Abstractions.TrxReportEngine.GetClassAndMethodName(TrxTestResult testResult)
   at Microsoft.Testing.Extensions.TrxReport.Abstractions.TrxReportEngine.AddResults(...)
   ...
Test host crashed and no test results could be recovered from the TRX streaming sidecar; the TRX is empty.
Test application process didn't exit gracefully, exit code is '-532462766'
host exit code: 7

(host exit code is $LASTEXITCODE of the launched executable, i.e. the parent test application; the child test host process is the one that crashed with -532462766.) The generated TRX has ResultSummary outcome="Failed", all counters 0, no UnitTest/UnitTestResult entries, and a single RunInfo error "Test host process pid: ... crashed.".

After: this branch's Debug build outputs (net10.0, built from a clean checkout of d7e51a5391efbcb89d14142f2654914fea1e8418), same input:

Test run summary: Passed!
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
host exit code: 0
ResultSummary outcome=Completed total=1 passed=1 failed=0
TestMethod className=UnknownClass name=bare display name
UnitTestResult testName=bare display name outcome=Passed duration=00:00:02.0000000

Nodes that do carry type information are unchanged. With E2E_SHAPE=all (default) the host publishes three nodes; the run exits 0 with 3 passed, ResultSummary Completed, and every UnitTestResult Passed with duration 00:00:02.0000000:

Node Properties besides final state + timing TestMethod/@className TestMethod/@name
bare none UnknownClass bare display name
fqtn TrxFullyQualifiedTypeNameProperty("Fqn.Namespace.FqnType") Fqn.Namespace.FqnType fqtn display name
ident TestMethodIdentifierProperty("asm", "My.Namespace", "MyType", "MyMethod", ...) My.Namespace.MyType MyMethod

Provenance. "Before" used the assemblies from the 2.5.3 package (product version 2.5.3+0b1d8b5f7b2186835e2cf26292f3478f7cca087e), referenced with <PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="2.5.3" />. "After" referenced this branch's Debug net10.0 outputs by HintPath (Microsoft.Testing.Platform, Microsoft.Testing.Extensions.TrxReport.Abstractions, Microsoft.Testing.Extensions.TrxReport); the SHA-256 of the scratch copies equals the build output:

Assembly SHA-256
Microsoft.Testing.Extensions.TrxReport.dll 909B1BC70B3E7F491C4F9380F6D93C876C7FFE98C3CE4B7FF45594E1BED9AA78
Microsoft.Testing.Extensions.TrxReport.Abstractions.dll A228EE2F75919F22427209CBA3C8040DFE386904B0F3858057E786F5C86D13F1
Microsoft.Testing.Platform.dll 5171E90CE36CF51730190E30C45B7D186F3301E6AA7128821C02A3EDE22DEE68
Scratch host source (Program.cs)
using Microsoft.Testing.Extensions;
using Microsoft.Testing.Extensions.TrxReport.Abstractions;
using Microsoft.Testing.Platform.Builder;
using Microsoft.Testing.Platform.Capabilities.TestFramework;
using Microsoft.Testing.Platform.Extensions.Messages;
using Microsoft.Testing.Platform.Extensions.TestFramework;

ITestApplicationBuilder builder = await TestApplication.CreateBuilderAsync(args);
builder.RegisterTestFramework(
    _ => new TestFrameworkCapabilities(new TrxReportCapability()),
    (_, _) => new StateOnlyTestFramework());
builder.AddTrxReportProvider();
using ITestApplication app = await builder.BuildAsync();
return await app.RunAsync();

internal sealed class TrxReportCapability : ITrxReportCapability
{
    bool ITrxReportCapability.IsSupported => true;

    void ITrxReportCapability.Enable()
    {
    }
}

// Mimics a framework (for example xUnit) that only reports a final state and timing.
// E2E_SHAPE=state-only publishes just that node; E2E_SHAPE=all (default) also publishes nodes
// carrying the optional type information, to show that their class names are unchanged.
internal sealed class StateOnlyTestFramework : ITestFramework, IDataProducer
{
    public string Uid => nameof(StateOnlyTestFramework);

    public string Version => "1.0.0";

    public string DisplayName => nameof(StateOnlyTestFramework);

    public string Description => nameof(StateOnlyTestFramework);

    public Type[] DataTypesProduced => [typeof(TestNodeUpdateMessage)];

    public Task<bool> IsEnabledAsync() => Task.FromResult(true);

    public Task<CreateTestSessionResult> CreateTestSessionAsync(CreateTestSessionContext context)
        => Task.FromResult(new CreateTestSessionResult { IsSuccess = true });

    public Task<CloseTestSessionResult> CloseTestSessionAsync(CloseTestSessionContext context)
        => Task.FromResult(new CloseTestSessionResult { IsSuccess = true });

    public async Task ExecuteRequestAsync(ExecuteRequestContext context)
    {
        DateTimeOffset start = new(2026, 9, 27, 12, 0, 0, TimeSpan.Zero);
        var timing = new TimingProperty(new TimingInfo(start, start.AddSeconds(2), TimeSpan.FromSeconds(2)));
        bool all = (Environment.GetEnvironmentVariable("E2E_SHAPE") ?? "all") == "all";

        await PublishAsync(context, "bare", "bare display name", new PropertyBag(PassedTestNodeStateProperty.CachedInstance, timing));

        if (all)
        {
            await PublishAsync(
                context,
                "fqtn",
                "fqtn display name",
                new PropertyBag(PassedTestNodeStateProperty.CachedInstance, timing, new TrxFullyQualifiedTypeNameProperty("Fqn.Namespace.FqnType")));

            await PublishAsync(
                context,
                "ident",
                "ident display name",
                new PropertyBag(
                    PassedTestNodeStateProperty.CachedInstance,
                    timing,
                    new TestMethodIdentifierProperty("asm", "My.Namespace", "MyType", "MyMethod", 0, [], "System.Void")));
        }

        context.Complete();
    }

    private async Task PublishAsync(ExecuteRequestContext context, string uid, string displayName, PropertyBag properties)
        => await context.MessageBus.PublishAsync(
            this,
            new TestNodeUpdateMessage(context.Request.Session.SessionUid, new TestNode { Uid = uid, DisplayName = displayName, Properties = properties }));
}

Validation and regression proof

Environment: Windows, PowerShell. The SDK pinned in global.json on the rebased branch, 12.0.100-alpha.1.26562.108, was installed by build.cmd into the git-ignored repo-local .dotnet folder. Source state: this branch at d7e51a5391efbcb89d14142f2654914fea1e8418, rebased onto main at ddb33a558d5200030053e571dba7a65de476f317. All green binaries were built from content identical to that commit, and the worktree is clean.

Build (warnings are errors in this repo):

.\build.cmd -projects "<repo>\test\UnitTests\Microsoft.Testing.Extensions.UnitTests\Microsoft.Testing.Extensions.UnitTests.csproj" -bl

Result: Build succeeded. with 0 Warning(s) and 0 Error(s), exit code 0. This builds Microsoft.Testing.Extensions.TrxReport and Microsoft.Testing.Extensions.TrxReport.Abstractions (netstandard2.0, net10.0, net11.0) and the unit-test project (net462, net472, net10.0, net11.0), in Debug. build.cmd passes -installWindowsSdk -restore -build to eng\build.ps1.

Tests (the test app is an MTP executable; run directly with the repo-local runtime):

$env:DOTNET_ROOT = "<repo>\.dotnet"; $env:DOTNET_MULTILEVEL_LOOKUP = "0"
$exe = "artifacts\bin\Microsoft.Testing.Extensions.UnitTests\Debug\<tfm>\Microsoft.Testing.Extensions.UnitTests.exe"
& $exe --filter "Name~OnlyStateAndTiming|Name~WithTestMethodIdentifier_ClassName" --progress off --output Normal
& $exe --filter "FullyQualifiedName~UnitTests.Trx|FullyQualifiedName~UnitTests.RetryTrxRegistrationBehaviorTests|FullyQualifiedName~TrxReport.Abstractions.UnitTests" --progress off --output Normal
Run TFM Exit code total succeeded failed skipped
4 new test cases net10.0 0 4 4 0 0
4 new test cases net11.0 0 4 4 0 0
4 new test cases net472 0 4 4 0 0
4 new test cases net462 0 4 4 0 0
All TRX-related test classes net10.0 0 401 390 0 11
All TRX-related test classes net11.0 0 401 390 0 11
All TRX-related test classes net472 0 396 396 0 0
All TRX-related test classes net462 0 396 396 0 0

The 11 skips on net10.0/net11.0 are pre-existing Linux-only link/merge tests (Merge_Linux*, MergeTool_Linux*, TemporarySibling_Linux*, TrxMerge_DanglingDestinationLink, TrxMerge_ExistingDestinationLink) reported as NotExecuted on Windows; none relate to this change.

Red/green (re-run on d7e51a5391efbcb89d14142f2654914fea1e8418). The new tests were kept, including their UnknownClass assertions. The two production files (TrxReportEngine.cs, TrxReportEngine.Results.cs) were restored to origin/main (ddb33a558d5200030053e571dba7a65de476f317) and touched so the incremental compile could not skip them. The tree was rebuilt with the same command (0 warnings, 0 errors, fresh Microsoft.Testing.Extensions.TrxReport.dll timestamp), and the 4 new cases were run on net10.0 with the filter above. Red, exit code 2:

failed TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName (273ms)
  System.InvalidOperationException: The test framework '' with UID '' does not support the 'ITrxReportCapability' leading to missing or incomplete TRX reports
failed Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName (274ms)
  System.InvalidOperationException: The test framework 'Fixture framework' with UID 'fixture.adapter' does not support the 'ITrxReportCapability' leading to missing or incomplete TRX reports
Test run summary: Failed!
  total: 4
  failed: 2
  succeeded: 2
  skipped: 0

This capture kept only the failure lines. On the original revision (a4eef71cb38972a0b0a9219652b02c5fd48bb640 against main at e92578e3a4c076150c8b4174fbe04bd5b9df91d6), the same red run showed the exception thrown from TrxReportEngine.GetClassAndMethodName (TrxReportEngine.Results.cs:284).

Green: both production files were restored to d7e51a5391efbcb89d14142f2654914fea1e8418, touched and rebuilt (0 warnings, 0 errors). The same 4 cases gave total: 4, failed: 0, succeeded: 4, skipped: 0, exit code 0. The worktree was clean afterwards.

Pitfall found on the original revision: right after restoring files, MSBuild's incremental compile can skip them because they keep old timestamps, so a "green" run can execute stale code. Both states above were touched before rebuilding.

Static checks: git diff --check origin/main..HEAD is clean; git diff --stat origin/main..HEAD is 6 files, 114 insertions, 7 deletions (3 C# files and one PACKAGE.md under src/Platform/, 2 test files); no PublicAPI.*.txt, .resx/.xlf, global.json or .lock.yml file is touched.

Test-value evidence

Tested revision: d7e51a5391efbcb89d14142f2654914fea1e8418. Counterfactual: the same tree with the two production files restored to origin/main (ddb33a558d5200030053e571dba7a65de476f317).

Test Contract Without the fix With the fix
TrxTests.TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName A node with only final state + timing generates a TRX: no warning, expected file name, outcome Completed, className UnknownClass, name = display name, duration 00:00:03.0000000 Fails: InvalidOperationException from GetClassAndMethodName Passes
TrxConsumerBehaviorTests.Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName The same node through the real consumer, streaming store and engine: UnknownClass, display-name method, duration 00:00:02.0000000, Completed, no output-device warning, one publication Fails: same exception Passes
TrxTests.TrxReportEngine_GenerateReportAsync_WithTestMethodIdentifier_ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType (2 DataRows) TrxFullyQualifiedTypeNameProperty wins; otherwise namespace + type name from the method identifier; method name MyMethod Passes (pins unchanged behavior, so it is expected to pass on both) Passes

Red: total: 4, failed: 2, succeeded: 2 (the two state-only tests). Green: total: 4, failed: 0, succeeded: 4. Existing tests of the unchanged failure paths (duplicate singleton properties: TrxPropertyProjectionTests.Extract_DuplicateSingleton_RejectsAmbiguousReportMetadata and TrxTestResultExtractorTests.Extract_DuplicateSingletonProperty_Throws; non-final state: TrxTestResultExtractorTests.Extract_InProgressState_ThrowsUnreachableException) are untouched and pass in the "All TRX-related test classes" runs.

Not validated / evidence gaps: CI results for d7e51a5391efbcb89d14142f2654914fea1e8418 were not available when this description was written. Linux and macOS were not run. The acceptance/integration test projects were not run (they need packed packages); the demonstration above runs the real host against build output, not packed packages. Only the TRX-related classes of the unit-test project were run, not the whole project. The second crash and controlled-failure reporting described under "Out of scope" are not covered.

Copilot AI balanced review requested due to automatic review settings November 13, 2026 16:46
@bradwilson

Copy link
Copy Markdown

However, this is very wrong and we probably should fail if TestMethodIdentifierProperty isn't present.

This is something we could also do on the xUnit.net side, though it's a little trickier because this is not necessarily TRX specific. Should we always publish one of these (with a synthetic name) even outside the scope of TRX?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change enabling TRX reporting of test results without type information.

No open issues 🎉

What changed in this PR

This PR makes Microsoft.Testing.Platform’s TRX reporter accept final test results without type information, addressing the source-less test scenario in #12436. It preserves existing identity precedence and leaves broader reporting-failure handling out of scope.

Changes:

  • Uses DefaultClassName instead of throwing when both identity properties are absent, retaining the display-name fallback for the method.
  • Updates API and package documentation to explain required, recommended, and optional properties.
  • Adds four test cases covering the fallback through report generation and streaming, plus unchanged identity precedence.

Confidence at a glance

🟡 TRX behavior, documentation, and tests — execution not independently verified

Test-value evidence: For revision a4eef71cb38972a0b0a9219652b02c5fd48bb640, the PR reports two fallback tests failing with production files restored from e92578e3a4c076150c8b4174fbe04bd5b9df91d6, then all four new cases passing with the fix. The two identity-precedence cases reportedly pass in both states. Tests were not rerun during this review; packed-package acceptance tests and Linux/macOS validation remain unperformed.

File Description
test/​UnitTests/​Microsoft.Testing.Extensions.UnitTests/​TrxTests.cs Tests fallback output and existing identity precedence.
test/​UnitTests/​Microsoft.Testing.Extensions.UnitTests/​TrxConsumerBehaviorTests.cs Tests the fallback through the streaming report lifecycle.
src/​Platform/​Microsoft.Testing.Extensions.TrxReport/​TrxReportEngine.Results.cs Replaces the missing-identity exception with the fallback.
src/​Platform/​Microsoft.Testing.Extensions.TrxReport/​TrxReportEngine.cs Defines the shared fallback class-name constant.
src/​Platform/​Microsoft.Testing.Extensions.TrxReport.Abstractions/​PACKAGE.md Summarizes property requirements and fallback behavior.
src/​Platform/​Microsoft.Testing.Extensions.TrxReport.Abstractions/​ITrxReportCapability.cs Updates XML documentation for optional identity metadata.

Review effort: Balanced

@github-actions

This comment has been minimized.

TrxReportEngine.GetClassAndMethodName threw InvalidOperationException when
a final test node had neither TrxFullyQualifiedTypeNameProperty nor
TestMethodIdentifierProperty. Test frameworks that only report a state (and
timing) could therefore not produce a TRX report at all.

The TRX schema requires a class name, so report "DefaultClassName" in that
case. This is the same last-resort name the VSTest TRX logger uses
(Converter.GetTestClassName). The method name keeps falling back to the node
display name. Nodes that have either property are unaffected, and validation
of the other properties (for example duplicate singleton properties) is
unchanged. No public API change.

Document the required (final state), recommended (type name or method
identifier) and optional TRX properties, including the fallback and the
duplicate-property failure, on ITrxReportCapability and in the
Abstractions package readme.

Related to #12436

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DefaultClassName looks like a real class in TRX consumers. UnknownClass makes it explicit that the test framework did not provide type information.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@Evangelink
Amaury Levé (Evangelink) force-pushed the dev/amauryleve/trx-fallback-class-name branch from a4eef71 to d7e51a5 Compare November 16, 2026 10:28
Copilot AI balanced review requested due to automatic review settings November 16, 2026 10:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The UnknownClass fallback conflicts with the described DefaultClassName contract and validation evidence, while the duplicate-property documentation overstates what the extractor enforces.

2 open issues

Review effort: Balanced


// The TRX schema requires a class name. When a test node has neither a fully qualified type name nor a test method
// identifier, use an explicit placeholder so consumers can tell that the type information is unknown.
private const string FallbackClassName = "UnknownClass";

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description has been reconciled: it now identifies UnknownClass, explains why DefaultClassName/VSTest parity was rejected, and records the post-rename demo and red/green evidence on d7e51a5. The current head 1380e72 is documentation-only.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 16, 2026
@github-actions github-actions Bot removed the state/needs-review Awaiting review from the team. label Nov 16, 2026
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review November 16, 2026 10:37
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12466

Compared merge base ddb33a558d5200030053e571dba7a65de476f317 with head d7e51a5391efbcb89d14142f2654914fea1e8418. Evidence applies to this head, not every intermediate commit.

Test-value evidence

All results are static predictions; no tests were executed and no paired execution records were available.

Contract Scenario / expected Test and decisive assertion Counterfactual Result
Node with only state (no type/method identity) produces a report instead of throwing State + timing only; className UnknownClass, name is the display name, duration 2s/3s, outcome Completed TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName and Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName: Assert.AreEqual("UnknownClass", className) Pre-change code throws InvalidOperationException, so both tests fail. Static prediction: killed Static prediction
Class name precedence: FQN property, else namespace + type DataRow(null, "My.Namespace.MyType") and DataRow("Fqn...", "Fqn...") ..._ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType: exact className and method name Swapping precedence fails one row. Likely passes before and after, since this was existing behavior Static prediction, characterization
Doc and PACKAGE.md updates, static modifier, comment change Non-executable n/a n/a Not applicable

Gaps: the FallbackClassName path when only TrxFullyQualifiedTypeNameProperty is present is covered by existing tests. The TrxDataConsumer warning that still uses TrxReportFrameworkDoesNotSupportTrxReportCapability is unchanged and still has its resource and test.

🧵 Parallel safety

PARALLEL-SAFETY RESULT
STATUS: PARTIAL

The specialist task was not launched in this run, so parallel-safety coverage is incomplete and this is not a clean result. A manual read of the changed tests found per-test MemoryFileStream/fixture instances and no new static or shared state. No parallelization configuration changed (config_count=0).

Per-test assessment

Pseudo-mutation evidence is a static prediction, not an executed mutation result.

Result Quality Test Notes How to improve
Pass A (90–100) new TrxTests.TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName Exact class, method name, duration and outcome assertions on the fallback path. None
Pass A (90–100) new TrxTests.TrxReportEngine_GenerateReportAsync_WithTestMethodIdentifier_ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType Both data rows assert exact className and method name. None
Pass A (90–100) new TrxConsumerBehaviorTests.Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName End-to-end consumer asserts exact className, name, duration, outcome and publication count. None

No inline findings: no verified, actionable defects.

This advisory review is heuristic and non-blocking. Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~28 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 16, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 16, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 16, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 16, 2026
@Evangelink

Copy link
Copy Markdown
Member Author

However, this is very wrong and we probably should fail if TestMethodIdentifierProperty isn't present.

This is something we could also do on the xUnit.net side, though it's a little trickier because this is not necessarily TRX specific. Should we always publish one of these (with a synthetic name) even outside the scope of TRX?

There is a little overhead to having unused properties, this is one of the reasons of the capabilities so you should be able to understand when or when not adding them. But if you always do, outside of the small memory overhead there is no problem.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 13:05
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/approved Proposal approved; ready for implementation. labels Nov 16, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 16, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12466

Compared merge base ddb33a558d5200030053e571dba7a65de476f317 with head 1380e72caeb9309c74f63490df8e36bbd0135407. Evidence applies to this head, not every intermediate commit.

Test-value evidence

All entries are static predictions; no paired execution records were available.

Contract Scenario / expected Test and decisive assertion Counterfactual Evidence
Final node with neither identity property no longer throws; class is UnknownClass, method is the display name State + timing only, expecting className UnknownClass and name equal to the display name TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_... and Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName assert className/name/duration Restoring the old InvalidOperationException makes both tests fail Static prediction: likely detected (inferred)
TrxFullyQualifiedTypeNameProperty wins, else namespace + type from the method identifier Two DataRows (null vs. FQN) ..._ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType asserts className and method name Swapping precedence fails the FQN row Static prediction: likely detected (inferred). Likely a pre/post-green characterization test.
XML docs, PACKAGE.md, comment, static modifier Non-executable or behavior-preserving n/a n/a Not applicable

Unverified: executed red/green was not available.

🧵 Parallel safety

STATUS: PARTIAL. The specialist task was not run in this pass, so coverage is incomplete and this is not a clean result. By inspection, the new tests use per-test MemoryFileStream/fixture instances and no static mutable state was introduced.

Per-test assessment

Pseudo-mutation evidence below consists of static predictions, not executed mutation results.

Result Quality Test Notes How to improve
Pass A (90–100) new TrxConsumerBehaviorTests.Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName Asserts class, name, duration, outcome and publication count end to end. None
Pass A (90–100) new TrxTests.TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName Exact values for class, method, duration and outcome. None
Pass A (90–100) new TrxTests.TrxReportEngine_GenerateReportAsync_WithTestMethodIdentifier_ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType Data rows cover both class-name sources with exact expectations. None

No inline findings met the publication gate. Existing-thread deduplication was not needed because there were no inline candidates.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~21 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 16, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the TRX reporting fallback that handles missing class and method metadata.

1 open issue
1 resolved since last review

Review effort: Balanced

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1683361

Result: inconclusive (evidence incomplete); no issue warranted.

  • Failure: DirectInvokerPreservesFrameworkFailuresAndReportsMalformedReflectionFailures (MSTestAdapter.PlatformServices.UnitTests, net10.0) failed in the Windows Debug leg, step "Test (unit tests only)". The assertion was "Expected object to refer to Task {Status=Running} because Timed out waiting for the owned assertion child", but the task was DelayPromiseWithCancellation {Status=Canceled}. The test ran for about 30s and the helper is DebugAssertionChildProbe.cs.
  • Assessment: The same test passed in the other collected results, including the Linux Release leg and recent main builds. The 30s duration points to a timeout waiting on a child process, so this looks like a timing or environment problem rather than a deterministic defect. Confidence: low to medium. It is not related to the TRX fallback class-name change in this PR as far as the evidence shows.
  • Not proven flaky: there is no fail-then-pass retry record for this test.
  • Other signals: NativeTests.Outcome and the WinUI packaged retry tests show failures in main history. They are not part of this PR's failing test step, so I am not attributing them to this PR.
  • Limits: history collection was incomplete (extraction-budget-exhausted), one artifact download failed, and no crash or hang diagnostics were found. I cannot claim there were no earlier occurrences.
  • Next step: re-run the Windows Debug leg. If the test fails again, investigate the child-process wait in DebugAssertionChildProbe.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~32 AIC · ⌖ ~1 AIC · ⊞ 10.1K

Brad Wilson (bradwilson) added a commit to xunit/xunit that referenced this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state/approved Proposal approved; ready for implementation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants