Repository navigation
Use a fallback class name in TRX reports for nodes without type info - #12466
Amaury Levé (Evangelink) wants to merge 3 commits into
Conversation
This is something we could also do on the xUnit.net side, though it's a little trickier because this is not necessarily TRX specific. Should we always publish one of these (with a synthetic name) even outside the scope of TRX? |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified for the change enabling TRX reporting of test results without type information.
No open issues 🎉
What changed in this PR
This PR makes Microsoft.Testing.Platform’s TRX reporter accept final test results without type information, addressing the source-less test scenario in #12436. It preserves existing identity precedence and leaves broader reporting-failure handling out of scope.
Changes:
- Uses
DefaultClassNameinstead of throwing when both identity properties are absent, retaining the display-name fallback for the method. - Updates API and package documentation to explain required, recommended, and optional properties.
- Adds four test cases covering the fallback through report generation and streaming, plus unchanged identity precedence.
Confidence at a glance
🟡 TRX behavior, documentation, and tests — execution not independently verified
Test-value evidence: For revision a4eef71cb38972a0b0a9219652b02c5fd48bb640, the PR reports two fallback tests failing with production files restored from e92578e3a4c076150c8b4174fbe04bd5b9df91d6, then all four new cases passing with the fix. The two identity-precedence cases reportedly pass in both states. Tests were not rerun during this review; packed-package acceptance tests and Linux/macOS validation remain unperformed.
| File | Description |
|---|---|
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/TrxTests.cs |
Tests fallback output and existing identity precedence. |
test/UnitTests/Microsoft.Testing.Extensions.UnitTests/TrxConsumerBehaviorTests.cs |
Tests the fallback through the streaming report lifecycle. |
src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxReportEngine.Results.cs |
Replaces the missing-identity exception with the fallback. |
src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxReportEngine.cs |
Defines the shared fallback class-name constant. |
src/Platform/Microsoft.Testing.Extensions.TrxReport.Abstractions/PACKAGE.md |
Summarizes property requirements and fallback behavior. |
src/Platform/Microsoft.Testing.Extensions.TrxReport.Abstractions/ITrxReportCapability.cs |
Updates XML documentation for optional identity metadata. |
Review effort: Balanced
This comment has been minimized.
This comment has been minimized.
TrxReportEngine.GetClassAndMethodName threw InvalidOperationException when a final test node had neither TrxFullyQualifiedTypeNameProperty nor TestMethodIdentifierProperty. Test frameworks that only report a state (and timing) could therefore not produce a TRX report at all. The TRX schema requires a class name, so report "DefaultClassName" in that case. This is the same last-resort name the VSTest TRX logger uses (Converter.GetTestClassName). The method name keeps falling back to the node display name. Nodes that have either property are unaffected, and validation of the other properties (for example duplicate singleton properties) is unchanged. No public API change. Document the required (final state), recommended (type name or method identifier) and optional TRX properties, including the fallback and the duplicate-property failure, on ITrxReportCapability and in the Abstractions package readme. Related to #12436 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DefaultClassName looks like a real class in TRX consumers. UnknownClass makes it explicit that the test framework did not provide type information. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
a4eef71 to
d7e51a5
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The UnknownClass fallback conflicts with the described DefaultClassName contract and validation evidence, while the duplicate-property documentation overstates what the extractor enforces.
2 open issues
Review effort: Balanced
|
|
||
| // The TRX schema requires a class name. When a test node has neither a fully qualified type name nor a test method | ||
| // identifier, use an explicit placeholder so consumers can tell that the type information is unknown. | ||
| private const string FallbackClassName = "UnknownClass"; |
There was a problem hiding this comment.
The description has been reconciled: it now identifies UnknownClass, explains why DefaultClassName/VSTest parity was rejected, and records the post-rename demo and red/green evidence on d7e51a5. The current head 1380e72 is documentation-only.
There was a problem hiding this comment.
🧪 Test quality review — PR #12466
Compared merge base ddb33a558d5200030053e571dba7a65de476f317 with head d7e51a5391efbcb89d14142f2654914fea1e8418. Evidence applies to this head, not every intermediate commit.
Test-value evidence
All results are static predictions; no tests were executed and no paired execution records were available.
| Contract | Scenario / expected | Test and decisive assertion | Counterfactual | Result |
|---|---|---|---|---|
| Node with only state (no type/method identity) produces a report instead of throwing | State + timing only; className UnknownClass, name is the display name, duration 2s/3s, outcome Completed |
TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName and Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName: Assert.AreEqual("UnknownClass", className) |
Pre-change code throws InvalidOperationException, so both tests fail. Static prediction: killed |
Static prediction |
| Class name precedence: FQN property, else namespace + type | DataRow(null, "My.Namespace.MyType") and DataRow("Fqn...", "Fqn...") |
..._ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType: exact className and method name |
Swapping precedence fails one row. Likely passes before and after, since this was existing behavior | Static prediction, characterization |
Doc and PACKAGE.md updates, static modifier, comment change |
Non-executable | n/a | n/a | Not applicable |
Gaps: the FallbackClassName path when only TrxFullyQualifiedTypeNameProperty is present is covered by existing tests. The TrxDataConsumer warning that still uses TrxReportFrameworkDoesNotSupportTrxReportCapability is unchanged and still has its resource and test.
🧵 Parallel safety
PARALLEL-SAFETY RESULT
STATUS: PARTIAL
The specialist task was not launched in this run, so parallel-safety coverage is incomplete and this is not a clean result. A manual read of the changed tests found per-test MemoryFileStream/fixture instances and no new static or shared state. No parallelization configuration changed (config_count=0).
Per-test assessment
Pseudo-mutation evidence is a static prediction, not an executed mutation result.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new TrxTests.TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName |
Exact class, method name, duration and outcome assertions on the fallback path. | None |
| Pass | A (90–100) | new TrxTests.TrxReportEngine_GenerateReportAsync_WithTestMethodIdentifier_ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType |
Both data rows assert exact className and method name. | None |
| Pass | A (90–100) | new TrxConsumerBehaviorTests.Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName |
End-to-end consumer asserts exact className, name, duration, outcome and publication count. | None |
No inline findings: no verified, actionable defects.
This advisory review is heuristic and non-blocking. Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~28 AIC · ⌖ ~1 AIC · ⊞ 8.1K
There is a little overhead to having unused properties, this is one of the reasons of the capabilities so you should be able to understand when or when not adding them. But if you always do, outside of the small memory overhead there is no problem. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12466
Compared merge base ddb33a558d5200030053e571dba7a65de476f317 with head 1380e72caeb9309c74f63490df8e36bbd0135407. Evidence applies to this head, not every intermediate commit.
Test-value evidence
All entries are static predictions; no paired execution records were available.
| Contract | Scenario / expected | Test and decisive assertion | Counterfactual | Evidence |
|---|---|---|---|---|
Final node with neither identity property no longer throws; class is UnknownClass, method is the display name |
State + timing only, expecting className UnknownClass and name equal to the display name |
TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_... and Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName assert className/name/duration |
Restoring the old InvalidOperationException makes both tests fail |
Static prediction: likely detected (inferred) |
TrxFullyQualifiedTypeNameProperty wins, else namespace + type from the method identifier |
Two DataRows (null vs. FQN) | ..._ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType asserts className and method name |
Swapping precedence fails the FQN row | Static prediction: likely detected (inferred). Likely a pre/post-green characterization test. |
XML docs, PACKAGE.md, comment, static modifier |
Non-executable or behavior-preserving | n/a | n/a | Not applicable |
Unverified: executed red/green was not available.
🧵 Parallel safety
STATUS: PARTIAL. The specialist task was not run in this pass, so coverage is incomplete and this is not a clean result. By inspection, the new tests use per-test MemoryFileStream/fixture instances and no static mutable state was introduced.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new TrxConsumerBehaviorTests.Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassName |
Asserts class, name, duration, outcome and publication count end to end. | None |
| Pass | A (90–100) | new TrxTests.TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodName |
Exact values for class, method, duration and outcome. | None |
| Pass | A (90–100) | new TrxTests.TrxReportEngine_GenerateReportAsync_WithTestMethodIdentifier_ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType |
Data rows cover both class-name sources with exact expectations. | None |
No inline findings met the publication gate. Existing-thread deduplication was not needed because there were no inline candidates.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~21 AIC · ⌖ ~1 AIC · ⊞ 8.1K
This comment has been minimized.
This comment has been minimized.
|
Final test-triage resolution — Azure Pipelines build 1683361 Result: inconclusive (evidence incomplete); no issue warranted.
|
…ierProperty when TRX is enabled


Pull request
Summary
Related to #12436 ("Which properties are required for Microsoft.Testing.Extensions.TrxReport?"). This PR does not close it; see "Out of scope" below.
Problem. A
TestNodethat has a final state but neitherTrxFullyQualifiedTypeNamePropertynorTestMethodIdentifierPropertymakesTrxReportEngine.GetClassAndMethodNamethrowInvalidOperationException("The test framework '...' with UID '...' does not support the 'ITrxReportCapability' leading to missing or incomplete TRX reports"). That is the shape reported by frameworks without source or type information (the issue notes xUnit.net supports test cases that don't come from source code at all). With the released 2.5.3 package the exception escapes report generation: the test host process crashes, the test application exits with code 7 and the TRX contains no test results (reproduced below).Change. When neither property is present, the class name is now the documented placeholder
UnknownClass, and the method name is the node's display name (already the behavior of the caller,testMethodName ?? testResultDisplayName).TrxReportEngine.cs/TrxReportEngine.Results.cs: newprivate const string FallbackClassName = "UnknownClass".GetClassAndMethodNamereturns(FallbackClassName, null)instead of throwing when there is no method identifier (and noTrxFullyQualifiedTypeNameProperty). The method no longer uses instance state, so it becamestatic, and the now-unused resourcesusingwas removed. TheTrxReportFrameworkDoesNotSupportTrxReportCapabilityresource is still used byTrxDataConsumerfor the "capability not supported" warning, so there are no.resx/.xlfchanges.TrxFullyQualifiedTypeNamePropertystill wins, then namespace + type name from the method identifier); the exceptions for a missing final state and for duplicate singleton properties.ITrxReportCapabilityXML documentation and theMicrosoft.Testing.Extensions.TrxReport.Abstractionspackage readme (PACKAGE.md) now say which properties are required (the final state), recommended (TrxFullyQualifiedTypeNamePropertyorTestMethodIdentifierProperty, with this fallback), optional (everything else), and that duplicates of consumed singleton properties make report generation fail, while unconsumed properties are ignored. This extends the documentation added in Document required and optional TRX report metadata #12458 and removes its now-stale sentence "Omitting both identity properties causes report generation to fail."mainatddb33a558d5200030053e571dba7a65de476f317. The original fix is now9d8106dabba1abd0e60b2dc7726232734a87fa24. After review,d7e51a5391efbcb89d14142f2654914fea1e8418renamed the placeholder fromDefaultClassNametoUnknownClassand removed the VSTest-parity claim from the docs. The demonstration and validation below were re-run ond7e51a5391efbcb89d14142f2654914fea1e8418.PublicAPI.*.txt, resources andglobal.jsonare untouched. No changelog entry:docs/Changelog-Platform.mdis updated in bulk at release time and recent TRX fix PRs don't touch it.Out of scope (why this is "Related to", not "Fixes"). The issue discussion also describes (1) a second crash: repeated results with the same UID that mix an explicit
TrxTestDefinitionNamewith an omitted one whose display-name fallback differs, and (2) a proposal to report genuine contract violations as controlled reporting failures (preserve recoverable results, mark the run unsuccessful, record the affected IDs inRunInfo, nonzero exit code). This PR implements the first part of the stated preference ("support source-less tests through a documented fallback") only.Design and decision record
This is a bug fix, not a new feature, so there is no separate design document; the rationale is here.
TestMethod/@classNameisuse="required"in the TRX schema (vstst.xsd, vendored copy; the XSD ships with Visual Studio and is not in this repo), so the report needs some class name, but a test node is not obliged to have one. The engine already accepts a missing method identifier wheneverTrxFullyQualifiedTypeNamePropertyis present (the method name falls back to the display name, which is how NUnit is reported today), so "neither property" was the only remaining hard failure. Failing the whole report, with exit code 7 and an empty TRX, for a node that has a valid final state is disproportionate.NOTEabove that display-name fallback ended with "However, this is very wrong and we probably should fail if TestMethodIdentifierProperty isn't present." This PR deliberately does the opposite for source-less nodes (the method-name fallback itself is unchanged), so I replaced that sentence with the new decision instead of leaving a comment that contradicts the code. Easy to restore if you would rather keep the original remark.UnknownClass. The name says what happened: the framework reported no type information, so the class is unknown. It does not read like a real default or a real type. It is a single identifier token, so consumers that splitclassNameon.into namespace and type still get a valid, unambiguous value. There was no existing constant or convention insrc/Platformto reuse.DefaultClassName(this PR's first revision). It is the initial value in the VSTest TRX logger'sConverter.GetTestClassName(microsoft/vstest@0f5ac692,Converter.cs#L593-L634). VSTest only returns it when the fully qualified name has no.or::and the test source file name is also unusable. The parity argument was therefore weak, and the name hides the fact that the type information is missing.ErrorUnknownClass). Source-less tests are legitimate; xUnit.net supports test cases that don't come from source code. TRX also has a realErroroutcome, so an error-sounding class name would suggest a failure that did not happen. Nothing is wrong with the result itself, only the grouping information is absent.DefaultClassNamefallback): environment-dependent, adds edge cases (empty or odd names), and looks like a real type.TrxFullyQualifiedTypeNameProperty, which keeps precedence (the issue author says they will do that framework-side).UnknownClassclass; each is still its own test definition with its own test id. A real type namedUnknownClassin the global namespace would share that class name; that is unlikely, and its tests would still have distinct test definitions. An empty fully qualified type name, or a method identifier with an empty type name, can still produce an empty class name (pre-existing and unchanged).Demonstration
Real composed product, no mocks: a minimal Microsoft Testing Platform host (source at the end of this section) whose test framework reports only a final state and timing, the shape from the issue, plus the real
Microsoft.Testing.Extensions.TrxReportassemblies. The same host source was built twice in a scratch project outside the repo (nothing of it is committed): once against the released package, once against this branch's build output.Before: released
Microsoft.Testing.Extensions.TrxReport2.5.3 from nuget.org (the shipped bits users have, not this PR's merge base):(
host exit codeis$LASTEXITCODEof the launched executable, i.e. the parent test application; the child test host process is the one that crashed with-532462766.) The generated TRX hasResultSummary outcome="Failed", all counters 0, noUnitTest/UnitTestResultentries, and a singleRunInfoerror "Test host process pid: ... crashed.".After: this branch's Debug build outputs (net10.0, built from a clean checkout of
d7e51a5391efbcb89d14142f2654914fea1e8418), same input:Nodes that do carry type information are unchanged. With
E2E_SHAPE=all(default) the host publishes three nodes; the run exits 0 with 3 passed,ResultSummaryCompleted, and everyUnitTestResultPassedwith duration00:00:02.0000000:TestMethod/@classNameTestMethod/@namebareUnknownClassbare display namefqtnTrxFullyQualifiedTypeNameProperty("Fqn.Namespace.FqnType")Fqn.Namespace.FqnTypefqtn display nameidentTestMethodIdentifierProperty("asm", "My.Namespace", "MyType", "MyMethod", ...)My.Namespace.MyTypeMyMethodProvenance. "Before" used the assemblies from the 2.5.3 package (product version
2.5.3+0b1d8b5f7b2186835e2cf26292f3478f7cca087e), referenced with<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="2.5.3" />. "After" referenced this branch's Debug net10.0 outputs byHintPath(Microsoft.Testing.Platform,Microsoft.Testing.Extensions.TrxReport.Abstractions,Microsoft.Testing.Extensions.TrxReport); the SHA-256 of the scratch copies equals the build output:Microsoft.Testing.Extensions.TrxReport.dll909B1BC70B3E7F491C4F9380F6D93C876C7FFE98C3CE4B7FF45594E1BED9AA78Microsoft.Testing.Extensions.TrxReport.Abstractions.dllA228EE2F75919F22427209CBA3C8040DFE386904B0F3858057E786F5C86D13F1Microsoft.Testing.Platform.dll5171E90CE36CF51730190E30C45B7D186F3301E6AA7128821C02A3EDE22DEE68Scratch host source (
Program.cs)Validation and regression proof
Environment: Windows, PowerShell. The SDK pinned in
global.jsonon the rebased branch,12.0.100-alpha.1.26562.108, was installed bybuild.cmdinto the git-ignored repo-local.dotnetfolder. Source state: this branch atd7e51a5391efbcb89d14142f2654914fea1e8418, rebased ontomainatddb33a558d5200030053e571dba7a65de476f317. All green binaries were built from content identical to that commit, and the worktree is clean.Build (warnings are errors in this repo):
Result:
Build succeeded.with0 Warning(s)and0 Error(s), exit code 0. This buildsMicrosoft.Testing.Extensions.TrxReportandMicrosoft.Testing.Extensions.TrxReport.Abstractions(netstandard2.0, net10.0, net11.0) and the unit-test project (net462, net472, net10.0, net11.0), in Debug.build.cmdpasses-installWindowsSdk -restore -buildtoeng\build.ps1.Tests (the test app is an MTP executable; run directly with the repo-local runtime):
The 11 skips on net10.0/net11.0 are pre-existing Linux-only link/merge tests (
Merge_Linux*,MergeTool_Linux*,TemporarySibling_Linux*,TrxMerge_DanglingDestinationLink,TrxMerge_ExistingDestinationLink) reported asNotExecutedon Windows; none relate to this change.Red/green (re-run on
d7e51a5391efbcb89d14142f2654914fea1e8418). The new tests were kept, including theirUnknownClassassertions. The two production files (TrxReportEngine.cs,TrxReportEngine.Results.cs) were restored toorigin/main(ddb33a558d5200030053e571dba7a65de476f317) and touched so the incremental compile could not skip them. The tree was rebuilt with the same command (0 warnings, 0 errors, freshMicrosoft.Testing.Extensions.TrxReport.dlltimestamp), and the 4 new cases were run on net10.0 with the filter above. Red, exit code 2:This capture kept only the failure lines. On the original revision (
a4eef71cb38972a0b0a9219652b02c5fd48bb640againstmainate92578e3a4c076150c8b4174fbe04bd5b9df91d6), the same red run showed the exception thrown fromTrxReportEngine.GetClassAndMethodName(TrxReportEngine.Results.cs:284).Green: both production files were restored to
d7e51a5391efbcb89d14142f2654914fea1e8418, touched and rebuilt (0 warnings, 0 errors). The same 4 cases gavetotal: 4, failed: 0, succeeded: 4, skipped: 0, exit code 0. The worktree was clean afterwards.Pitfall found on the original revision: right after restoring files, MSBuild's incremental compile can skip them because they keep old timestamps, so a "green" run can execute stale code. Both states above were touched before rebuilding.
Static checks:
git diff --check origin/main..HEADis clean;git diff --stat origin/main..HEADis 6 files, 114 insertions, 7 deletions (3 C# files and onePACKAGE.mdundersrc/Platform/, 2 test files); noPublicAPI.*.txt,.resx/.xlf,global.jsonor.lock.ymlfile is touched.Test-value evidence
Tested revision:
d7e51a5391efbcb89d14142f2654914fea1e8418. Counterfactual: the same tree with the two production files restored toorigin/main(ddb33a558d5200030053e571dba7a65de476f317).TrxTests.TrxReportEngine_GenerateReportAsync_WithOnlyStateAndTiming_UsesFallbackClassNameAndDisplayNameAsMethodNameCompleted,classNameUnknownClass,name= display name, duration00:00:03.0000000InvalidOperationExceptionfromGetClassAndMethodNameTrxConsumerBehaviorTests.Session_NodeWithOnlyStateAndTimingStreamsAndReportsFallbackClassNameUnknownClass, display-name method, duration00:00:02.0000000,Completed, no output-device warning, one publicationTrxTests.TrxReportEngine_GenerateReportAsync_WithTestMethodIdentifier_ClassNameIsFullyQualifiedTypeNameElseNamespaceAndType(2DataRows)TrxFullyQualifiedTypeNamePropertywins; otherwise namespace + type name from the method identifier; method nameMyMethodRed:
total: 4, failed: 2, succeeded: 2(the two state-only tests). Green:total: 4, failed: 0, succeeded: 4. Existing tests of the unchanged failure paths (duplicate singleton properties:TrxPropertyProjectionTests.Extract_DuplicateSingleton_RejectsAmbiguousReportMetadataandTrxTestResultExtractorTests.Extract_DuplicateSingletonProperty_Throws; non-final state:TrxTestResultExtractorTests.Extract_InProgressState_ThrowsUnreachableException) are untouched and pass in the "All TRX-related test classes" runs.Not validated / evidence gaps: CI results for
d7e51a5391efbcb89d14142f2654914fea1e8418were not available when this description was written. Linux and macOS were not run. The acceptance/integration test projects were not run (they need packed packages); the demonstration above runs the real host against build output, not packed packages. Only the TRX-related classes of the unit-test project were run, not the whole project. The second crash and controlled-failure reporting described under "Out of scope" are not covered.