Repository navigation
Description
Summary
Open an umbrella discussion to track LLM- and agent-efficiency improvements for Microsoft.Testing.Platform (MTP) and MSTest output.
LLMs and coding agents (Copilot, Claude Code, Cursor, Codex, Aider, …) are increasingly the primary readers of dotnet test output. Each token of noise we emit translates into context-window pressure, slower agent loops, and more LLM cost. The repo already has scaffolding for this (LLMEnvironmentDetector, LLM-aware --show-stdout/--show-stderr defaults via #8771, --list-tests json via #8280, --no-ansi/--no-progress flags via #7647/#7649), but several large wins are still missing.
This RFC-style issue collects the candidate improvements and links to focused sub-issues so they can be discussed, prioritized, and shipped individually.
Motivation
A typical agent flow today looks like:
- Agent runs
dotnet test(ordotnet run --project ...for MTP). - The test host prints a banner, runtime info, a live progress region, and per-test outcome lines (~all in color).
- On failure, each failing test prints message + expected/actual + stack trace including ~10 MTP/MSTest plumbing frames.
- The agent must reread the whole transcript to figure out which tests failed and where.
Symptoms an agent sees today, all of which cost tokens:
- ANSI color escape codes are emitted in non-LLM CI runs unless
NO_COLOR-equivalent flags are set. The de-facto industry standard env varNO_COLORis not honored by name. - Stack traces include the full MTP plumbing chain (
Microsoft.Testing.Platform.Hosts.*,Microsoft.Testing.Platform.MSTest.*,Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*, …). User code is only a handful of frames; the rest is noise. - There is no compact "what failed and where" footer, so the agent has to re-parse the whole log.
- There is no first-class machine-readable test-results output (TRX is XML and verbose;
--list-tests jsononly covers discovery; HTML report is not parseable). Each failure block is paragraph-style text. --show-stdout/--show-stderrhelp text still claims default isAlleven though it flips toFailedin LLM mode (tracked at Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772, still open).dotnet test/MSBuild wrapping can hide MTP's failure detail behindTestingPlatformShowTestsFailure=false(see No failing test details with MTP and dotnet test support with xunit v3 #5188).
Detailed proposals
Each item below should be filed as a separate issue (linked when filed) so it can be designed, sized, and shipped independently.
Output: noise reduction
-
Honor
NO_COLORenv var natively (industry standard, see https://no-color.org). Treat any non-emptyNO_COLORlike--no-ansi.--ansi on|off|autoshould still take precedence.
→ Honor NO_COLOR environment variable in MTP terminal output #8825 -
Filter MTP/MSTest internal stack frames in the terminal failure output. Hide frames whose declaring type lives under
Microsoft.Testing.Platform.*(non-public namespaces) orMicrosoft.VisualStudio.TestPlatform.MSTestAdapter.*. Keep user-code frames. Apply at the rendering layer (FormatStackTrace/AppendStackFrameinTerminalTestReporter.Formatting.cs) so raw exception data flowing to TRX/HTML/AzDO reports is untouched. Default-on in LLM mode, opt-in flag (--full-stack-trace) for humans who want everything.
→ Filter MTP/MSTest internal stack frames from terminal failure output #8826 — closed as not planned (decided against: internal MTP/MSTest frames are kept for debuggability) -
Deduplicate identical failures across tests. Group failures whose
(exception type + first N stack frames + first line of message)are identical; print details once, reference subsequent failures by count + list of test ids. Reduces N×output bloat when a shared helper breaks.
→ sub-issue: to be filed -
Crash/hang dump output volume — verify Excessive output when crashdump kicks in #7045 fix scope; sometimes a dump still prints multi-screen native stacks before the test summary.
Output: machine-readable / structured
-
Microsoft.Testing.Extensions.JsonReportNDJSON event stream (new extension, modeled onTrxReport/HtmlReport/AzureDevOpsReport). One JSON object per test event (started/completed) + a final summary. Stable schema documented inPACKAGE.md. Inspired by Jest's--json, pytest-json-report, and Vitest's JSON reporter.
→ Add JSON / NDJSON test report extension (Microsoft.Testing.Extensions.JsonReport) #8828 -
Compact end-of-run failure index footer (sentinel-delimited block listing only
Test FQN | first-line error | file:lineper failure). Agents can grep that block alone instead of re-reading the full log. Default-on in LLM mode only, no CLI flag to avoid help/info-snapshot churn for the first revision.
→ Emit compact end-of-run failure index for LLM/agent consumption #8827 -
OpenTelemetry sanitization — sanitize raw paths/PII/stack traces before export ([MTP Diagnostic Analysis] OpenTelemetry exports raw file paths, stdout/stderr, and stack traces without filtering #8411 already open).
Input: agent ergonomics
-
Fix
--show-stdout/--show-stderrhelp text to honestly describe the LLM-mode-aware default (Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772 already open). Needs XLF regen + help/info acceptance updates. -
Reduce default banner/preamble in LLM mode. "Microsoft (R) Testing Platform vX.Y.Z" + runtime info + per-extension banners cost tokens before any test even runs. In LLM mode, skip or compress to a single line.
-
--help --output json/--info --output jsonfor machine-readable help/info. Agents can query for the option they need without scanning ~160 lines of text. -
Stable error codes for common misconfigurations. Today, errors like "option X requires Y" are English-only — give them codes (e.g.
MTP1001) so agents can match without prompt-engineering.
dotnet test / MSBuild wrapping
- Agent-friendly
dotnet testdefaults / documentation.TestingPlatformShowTestsFailure=falseby default inMicrosoft.Testing.Platform.MSBuild.targetsmeans agents only see a pointer to a log file unless they opt in. Flip default totruewhen an LLM env is detected (or document the env var to flip it). Cross-link No failing test details with MTP and dotnet test support with xunit v3 #5188.
Documentation
-
Public doc page: "Microsoft.Testing.Platform for AI agents." Document
LLMEnvironmentDetectorbehavior, detected env vars,NO_COLOR, recommended flags, JSON report extension. -
AGENTS.mdat repo root referencing.github/copilot-instructions.md(industry convention emerging in 2025).
Drawbacks
- Each item changes some part of the terminal transcript and risks breaking log-scraping CI consumers. Mitigations: gate behavioral changes to LLM-mode-only by default, expose opt-in/opt-out flags for humans, and keep raw report data (TRX/HTML/AzDO) untouched.
- Adding a new JSON report extension grows the package surface and the maintenance footprint.
- Stack-trace filtering can hide a frame that's actually relevant to extension authors debugging MTP itself; needs an opt-out path.
Alternatives
- Do nothing and rely on agents to learn to ignore the noise. Trades user tokens/cost.
- Wait for the .NET SDK to standardize an agent-friendly test output. Doesn't help today.
Compatibility
Not a breaking change if behavioral defaults stay LLM-mode-gated and human-mode output is unchanged.
Unresolved questions
- Should the JSON report extension be packaged separately or rolled into an existing one?
- Should NO_COLOR support also imply
--no-progress, or stay color-only as the spec says? - Should the failure-index footer be a flag (
--failure-index/--no-failure-index) or env var, or strictly LLM-mode-gated for v1?
References
src/Platform/Microsoft.Testing.Platform/Helpers/LLMEnvironmentDetector.cssrc/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.cs(LLM-aware behavior switches)src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/ExceptionFlattener.cs(stack trace passthrough)src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.Formatting.cs(stack-frame rendering)- Related: Add "no progress" and "no ANSI" environment variables #7647, Avoid ANSI and progress output when running in LLM environment #7649, [Lean Squad] feat(fv): Task 1 — expand TARGETS.md to 17 targets; add LLMEnvironmentDetector research #8084, Add --list-tests json for machine-readable test discovery output #8280, [MTP Diagnostic Analysis] OpenTelemetry exports raw file paths, stdout/stderr, and stack traces without filtering #8411, Suppress banner and default stdout/stderr to
failedin LLM environments #8771, Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772, No failing test details with MTP and dotnet test support with xunit v3 #5188, Excessive output when crashdump kicks in #7045 - External: https://no-color.org, Jest
--json, pytest-json-report, Vitest reporters
Filed by Amaury Levé (@Evangelink) based on an analysis of the MTP/MSTest output surfaces from an agent-consumer perspective.
Activity
Evangelink commentedon Jun 24, 2026
Adding an energy-efficiency lens to these proposals, since reducing LLM agent output volume has a measurable energy dimension.
The energy cost of noisy test output
LLM-based agents are increasingly the primary consumers of
dotnet testoutput. Each extra token in the output:Rough estimate: a typical failing
dotnet testrun with 5 failed tests currently outputs ~200 lines including ANSI codes, banners, runtime info, and full stack traces. Modern LLMs running at ~10 ms/token decode cost ~100 ms to process that block. At millions of CI runs per day across the .NET ecosystem, even a 50 % output reduction would translate to material token-processing energy savings.Prioritised by energy impact
Ordering the RFC proposals by estimated energy reduction per agent invocation:
Microsoft.Testing.Extensions.JsonReportNDJSON (#8828)NO_COLOR(#8825, ✅ done)Stack-frame filtering: a note on implementation energy
The proposal to filter in
FormatStackTrace/AppendStackFrameat the rendering layer (not at the exception capture layer) is the right call from an energy perspective too: it means the filtering happens once, at print time, rather than on every code path that might consume the exception (TRX writer, HTML report, AzDO reporter). No double-processing cost.Suggestion: measure output size as a CI health metric
Once the LLM-mode changes land, it would be worth tracking raw byte count of
dotnet testoutput for a representative test suite as a CI signal — similar to how bundle size is tracked for web apps. A sudden output-size increase flags a noisy new message before it ships. This would give the project an ongoing "output energy budget" anchor.Green Software Foundation framing
These proposals collectively address Demand Shaping (emit less data = less downstream processing demand) and Hardware Efficiency (agent GPU inference cost per run). The compact failure-index footer in particular is a good example of shaping demand at the source — agents only need to read one small block instead of scanning the whole log.
Add this agentic workflows to your repo
To install this agentic workflow, run
Evangelink commentedon Jul 14, 2026
🔄 State refresh — sub-issue reconciliation
Reviewed the linked sub-issues and the current codebase state. Summary of where each RFC item stands:
✅ Completed / merged
NO_COLOR— Honor NO_COLOR environment variable in MTP terminal output #8825 ✅ closed (completed).NO_COLORhandling is in the terminal output device.--show-stdout/--show-stderrhelp text — Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772 ✅ closed (completed).failedin LLM environments #8771 ✅ merged (banner suppression +Failedstdout/stderr default in LLM envs).--list-tests json(prior art) — Add --list-tests json for machine-readable test discovery output #8280 ✅ merged.🚫 Not planned (state corrected in checklist)
🟡 Still open
dotnet test/ MSBuild defaults — cross-links No failing test details with MTP and dotnet test support with xunit v3 #5188 (open).📥 Not yet filed (no sub-issue)
--help --output json/--info --output json.AGENTS.mdreferencing.github/copilot-instructions.md— note: a rootAGENTS.mdexists today, but it is thegh-copilot-curate-managed skills index, not the copilot-instructions pointer this item asks for, so it remains unchecked.Checkbox marks in the body were already accurate for everything except #8826, which is now annotated.
Evangelink commentedon Oct 15, 2026
Measurement follow-up: small format savings, stronger evidence-fidelity findings
A local pilot against TestFx revision
6c6b1f474acbb6ab1bdbdec73fa32568f12dfcb5does not yet justify a new default footer or another JSON format solely for token-cost reduction.What ran
gpt-6.1-sol.The toolchain followed the pinned dotnet/skills evaluation infrastructure: Vally 0.14.0, Copilot SDK 1.0.11 and its nested CLI 1.0.80. Product captures used Windows x64, net9.0, Release, MTP
2.6.0-dev, CTRF1.0.0-dev, and repository SDK12.0.100-alpha.1.26509.101. Package, cache, output-DLL and actual loaded-path provenance was checked.Measured pilot
Each synthetic arm received the same run metadata, prompt, exact-result grader and optional full-detail transcript. These are file-first triage observations, not live streaming or clipped-terminal measurements.
Input usage includes repeated/cached context; provider cost is the reported
nano-aiumetric, not a USD or energy estimate. The primary run reported 2,170,849 input tokens, including 1,534,979 cached-read tokens. Raw bytes, input-token totals and billed/provider cost are different quantities.One footer-first trial had two recovered read-path mistakes, retained in the totals. The two-case real-output calibration did not reproduce a consistent saving: footer-first cost was -0.2%, while CTRF-query was +7.9%, with two recovered read-path mistakes in one query trial. Neither that increase nor the small synthetic reductions establish an intrinsic format effect. A preliminary Windows session-filesystem/tool defect was isolated; its incomplete attempt was excluded from comparisons, not selectively merged into successful trials.
More actionable findings
My recommendation is to keep current defaults, reuse targeted queries over existing reports when those reports are already needed, and prioritize diagnostic fidelity and cheap retrieval. Test #8827 with genuinely large/clipped logs before considering a default. Justify #8828 separately through live-consumer/time-to-first-action requirements.
Evidence limits
This is one model, one repeat, eight synthetic cases and two controlled real-output calibrations. It measures exact triage facts, not code-repair success, all agent hosts, other operating systems, live streaming, or servicing branches. There is no measured energy/carbon result. The harness, CSV and raw trajectories are currently retained locally, not committed or hosted as a public benchmark artifact; the figures above are bounded pilot observations rather than an independently downloadable benchmark.
Primary run:
ec9c14a6-002b-4a7c-8598-681685d6618c. Real calibration:744863c9-d6e4-4d75-b63e-8c2d8af419dd. No shipping implementation or issue-state change accompanied the evaluation.Posted by GitHub Copilot at the maintainer's request.
Evangelink commentedon Nov 9, 2026
Closing this RFC as completed as an investigation and decision record, not as a claim that every proposed feature shipped. The current checklist combines delivered work, declined designs, and unmeasured hypotheses; the remaining actionable work is now tracked separately.
Decisions and current state
Focused remaining work
I am working on the first two follow-ups. They remain open until their actual changes and validation are ready; closing this umbrella does not mark them implemented.
Evidence and compatibility
The independent Claude Opus 5.5, Gemini 4 Argon and GPT-6.1 Sol reviews agreed against implementing the checklist wholesale. These are source/design assessments, not a new runtime benchmark. The earlier single-model pilot (#8824 (comment)) had all 54 exact triage grades pass, but small/mixed provider-cost changes and no clipped-log, repair, live-streaming or energy measurement. Output bytes, input tokens, provider cost and energy are different quantities; the earlier automated energy estimates are not measured evidence.
Shipped MTP 2.5.2 defaults captured stdout/stderr to Failed in detected agent environments and All otherwise. The universal Failed default, non-cursor summary reordering and CTRF graduation are staged under unreleased 3.0 in the changelog, not already shipped 2.x behavior. Progress is independently controlled. New defaults still need explicit compatibility and consumer evidence, even when gated on agent detection.