Skip to content

[RFC] Agent/LLM-efficient test output for Microsoft.Testing.Platform #8824

Closed

Description

Summary

Open an umbrella discussion to track LLM- and agent-efficiency improvements for Microsoft.Testing.Platform (MTP) and MSTest output.

LLMs and coding agents (Copilot, Claude Code, Cursor, Codex, Aider, …) are increasingly the primary readers of dotnet test output. Each token of noise we emit translates into context-window pressure, slower agent loops, and more LLM cost. The repo already has scaffolding for this (LLMEnvironmentDetector, LLM-aware --show-stdout/--show-stderr defaults via #8771, --list-tests json via #8280, --no-ansi/--no-progress flags via #7647/#7649), but several large wins are still missing.

This RFC-style issue collects the candidate improvements and links to focused sub-issues so they can be discussed, prioritized, and shipped individually.

Motivation

A typical agent flow today looks like:

  1. Agent runs dotnet test (or dotnet run --project ... for MTP).
  2. The test host prints a banner, runtime info, a live progress region, and per-test outcome lines (~all in color).
  3. On failure, each failing test prints message + expected/actual + stack trace including ~10 MTP/MSTest plumbing frames.
  4. The agent must reread the whole transcript to figure out which tests failed and where.

Symptoms an agent sees today, all of which cost tokens:

  • ANSI color escape codes are emitted in non-LLM CI runs unless NO_COLOR-equivalent flags are set. The de-facto industry standard env var NO_COLOR is not honored by name.
  • Stack traces include the full MTP plumbing chain (Microsoft.Testing.Platform.Hosts.*, Microsoft.Testing.Platform.MSTest.*, Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*, …). User code is only a handful of frames; the rest is noise.
  • There is no compact "what failed and where" footer, so the agent has to re-parse the whole log.
  • There is no first-class machine-readable test-results output (TRX is XML and verbose; --list-tests json only covers discovery; HTML report is not parseable). Each failure block is paragraph-style text.
  • --show-stdout / --show-stderr help text still claims default is All even though it flips to Failed in LLM mode (tracked at Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772, still open).
  • dotnet test/MSBuild wrapping can hide MTP's failure detail behind TestingPlatformShowTestsFailure=false (see No failing test details with MTP and dotnet test support with xunit v3 #5188).

Detailed proposals

Each item below should be filed as a separate issue (linked when filed) so it can be designed, sized, and shipped independently.

Output: noise reduction

  • Honor NO_COLOR env var natively (industry standard, see https://no-color.org). Treat any non-empty NO_COLOR like --no-ansi. --ansi on|off|auto should still take precedence.
    → Honor NO_COLOR environment variable in MTP terminal output #8825

  • Filter MTP/MSTest internal stack frames in the terminal failure output. Hide frames whose declaring type lives under Microsoft.Testing.Platform.* (non-public namespaces) or Microsoft.VisualStudio.TestPlatform.MSTestAdapter.*. Keep user-code frames. Apply at the rendering layer (FormatStackTrace/AppendStackFrame in TerminalTestReporter.Formatting.cs) so raw exception data flowing to TRX/HTML/AzDO reports is untouched. Default-on in LLM mode, opt-in flag (--full-stack-trace) for humans who want everything.
    → Filter MTP/MSTest internal stack frames from terminal failure output #8826 — closed as not planned (decided against: internal MTP/MSTest frames are kept for debuggability)

  • Deduplicate identical failures across tests. Group failures whose (exception type + first N stack frames + first line of message) are identical; print details once, reference subsequent failures by count + list of test ids. Reduces N×output bloat when a shared helper breaks.
    → sub-issue: to be filed

  • Crash/hang dump output volume — verify Excessive output when crashdump kicks in #7045 fix scope; sometimes a dump still prints multi-screen native stacks before the test summary.

Output: machine-readable / structured

Input: agent ergonomics

  • Fix --show-stdout/--show-stderr help text to honestly describe the LLM-mode-aware default (Update --show-stdout / --show-stderr help text to reflect LLM-aware default #8772 already open). Needs XLF regen + help/info acceptance updates.

  • Reduce default banner/preamble in LLM mode. "Microsoft (R) Testing Platform vX.Y.Z" + runtime info + per-extension banners cost tokens before any test even runs. In LLM mode, skip or compress to a single line.

  • --help --output json / --info --output json for machine-readable help/info. Agents can query for the option they need without scanning ~160 lines of text.

  • Stable error codes for common misconfigurations. Today, errors like "option X requires Y" are English-only — give them codes (e.g. MTP1001) so agents can match without prompt-engineering.

dotnet test / MSBuild wrapping

  • Agent-friendly dotnet test defaults / documentation. TestingPlatformShowTestsFailure=false by default in Microsoft.Testing.Platform.MSBuild.targets means agents only see a pointer to a log file unless they opt in. Flip default to true when an LLM env is detected (or document the env var to flip it). Cross-link No failing test details with MTP and dotnet test support with xunit v3 #5188.

Documentation

  • Public doc page: "Microsoft.Testing.Platform for AI agents." Document LLMEnvironmentDetector behavior, detected env vars, NO_COLOR, recommended flags, JSON report extension.

  • AGENTS.md at repo root referencing .github/copilot-instructions.md (industry convention emerging in 2025).

Drawbacks

  • Each item changes some part of the terminal transcript and risks breaking log-scraping CI consumers. Mitigations: gate behavioral changes to LLM-mode-only by default, expose opt-in/opt-out flags for humans, and keep raw report data (TRX/HTML/AzDO) untouched.
  • Adding a new JSON report extension grows the package surface and the maintenance footprint.
  • Stack-trace filtering can hide a frame that's actually relevant to extension authors debugging MTP itself; needs an opt-out path.

Alternatives

  • Do nothing and rely on agents to learn to ignore the noise. Trades user tokens/cost.
  • Wait for the .NET SDK to standardize an agent-friendly test output. Doesn't help today.

Compatibility

Not a breaking change if behavioral defaults stay LLM-mode-gated and human-mode output is unchanged.

Unresolved questions

  • Should the JSON report extension be packaged separately or rolled into an existing one?
  • Should NO_COLOR support also imply --no-progress, or stay color-only as the spec says?
  • Should the failure-index footer be a flag (--failure-index / --no-failure-index) or env var, or strictly LLM-mode-gated for v1?

References

Filed by Amaury Levé (@Evangelink) based on an analysis of the MTP/MSTest output surfaces from an agent-consumer perspective.

Activity

  1. Evangelink commented on Jun 24, 2026

    @Evangelink
    MemberAuthor

    Adding an energy-efficiency lens to these proposals, since reducing LLM agent output volume has a measurable energy dimension.

    The energy cost of noisy test output

    LLM-based agents are increasingly the primary consumers of dotnet test output. Each extra token in the output:

    1. Consumes GPU energy during the LLM's decode pass — token generation is the dominant cost in transformer inference.
    2. Increases context window pressure, potentially forcing a re-prompt (another full-context inference) when the window fills.
    3. Delays agent iteration loops — more tokens to process = more wall-clock time = more energy per CI/CD cycle.

    Rough estimate: a typical failing dotnet test run with 5 failed tests currently outputs ~200 lines including ANSI codes, banners, runtime info, and full stack traces. Modern LLMs running at ~10 ms/token decode cost ~100 ms to process that block. At millions of CI runs per day across the .NET ecosystem, even a 50 % output reduction would translate to material token-processing energy savings.

    Prioritised by energy impact

    Ordering the RFC proposals by estimated energy reduction per agent invocation:

    Proposal Token Reduction Energy Impact Notes
    Filter MTP/MSTest internal stack frames (#8826) HIGH (~60–80 % of stack trace token count) HIGH Stack frames are long strings; filtering plumbing frames is the biggest single win
    Compact failure-index footer (#8827) MEDIUM MEDIUM Allows agents to skip reading the body entirely — whole-context reduction
    Microsoft.Testing.Extensions.JsonReport NDJSON (#8828) MEDIUM MEDIUM Machine-readable = no redundant re-parsing; compact format
    Honor NO_COLOR (#8825, ✅ done) LOW-MEDIUM LOW-MEDIUM ANSI escape codes add ~10–15 % to raw byte count in failure output
    Reduce banner in LLM mode (✅ done) LOW LOW Banner is small relative to test output

    Stack-frame filtering: a note on implementation energy

    The proposal to filter in FormatStackTrace/AppendStackFrame at the rendering layer (not at the exception capture layer) is the right call from an energy perspective too: it means the filtering happens once, at print time, rather than on every code path that might consume the exception (TRX writer, HTML report, AzDO reporter). No double-processing cost.

    Suggestion: measure output size as a CI health metric

    Once the LLM-mode changes land, it would be worth tracking raw byte count of dotnet test output for a representative test suite as a CI signal — similar to how bundle size is tracked for web apps. A sudden output-size increase flags a noisy new message before it ships. This would give the project an ongoing "output energy budget" anchor.

    Green Software Foundation framing

    These proposals collectively address Demand Shaping (emit less data = less downstream processing demand) and Hardware Efficiency (agent GPU inference cost per run). The compact failure-index footer in particular is a good example of shaping demand at the source — agents only need to read one small block instead of scanning the whole log.

    🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Efficiency Improver workflow. · 2.3K AIC · ⌖ 40.6 AIC · ⊞ 58.6K · [◷]( · ◷)

    Add this agentic workflows to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/efficiency-improver.md@main
    
  2. Evangelink commented on Jul 14, 2026

    @Evangelink
    MemberAuthor

    🔄 State refresh — sub-issue reconciliation

    Reviewed the linked sub-issues and the current codebase state. Summary of where each RFC item stands:

    ✅ Completed / merged

    🚫 Not planned (state corrected in checklist)

    🟡 Still open

    📥 Not yet filed (no sub-issue)

    • Deduplicate identical failures across tests.
    • --help --output json / --info --output json.
    • Stable error codes for common misconfigurations.
    • Public doc page "Microsoft.Testing.Platform for AI agents."
    • AGENTS.md referencing .github/copilot-instructions.md — note: a root AGENTS.md exists today, but it is the gh-copilot-curate-managed skills index, not the copilot-instructions pointer this item asks for, so it remains unchecked.

    Checkbox marks in the body were already accurate for everything except #8826, which is now annotated.

  3. Evangelink commented on Oct 15, 2026

    @Evangelink
    MemberAuthor

    Measurement follow-up: small format savings, stronger evidence-fidelity findings

    A local pilot against TestFx revision 6c6b1f474acbb6ab1bdbdec73fa32568f12dfcb5 does not yet justify a new default footer or another JSON format solely for token-cost reduction.

    What ran

    • 48 model-backed triage trials: eight synthetic scenarios, six output/access views, one repeat and one worker, using gpt-6.1-sol.
    • Six separate calibration trials: two captured outputs from the actual packed MTP/CTRF host, compared as terminal, prototype footer-first and CTRF failure query.
    • All 54 exact triage grades passed. Saved answers were rechecked, required input hashes matched, and all expected trial slots were accounted for.
    • Six real-host scenarios also had CTRF-off/on process measurements: one warmup and five measured samples per mode, giving 60 measured launches. The consumer used a controlled framework, not the MSTest adapter or an automated repair task.

    The toolchain followed the pinned dotnet/skills evaluation infrastructure: Vally 0.14.0, Copilot SDK 1.0.11 and its nested CLI 1.0.80. Product captures used Windows x64, net9.0, Release, MTP 2.6.0-dev, CTRF 1.0.0-dev, and repository SDK 12.0.100-alpha.1.26509.101. Package, cache, output-DLL and actual loaded-path provenance was checked.

    Measured pilot

    Each synthetic arm received the same run metadata, prompt, exact-result grader and optional full-detail transcript. These are file-first triage observations, not live streaming or clipped-terminal measurements.

    Initial view Correct Mean input tokens Input change Reported provider-cost change
    Terminal 8/8 48,306 baseline baseline
    Terminal + prototype footer 8/8 43,357 -10.2% -2.1%
    Prototype footer-first 8/8 51,519 +6.7% +1.3%
    CTRF-shaped failure query 8/8 41,633 -13.8% -2.8%
    Prototype full NDJSON 8/8 44,910 -7.0% +4.2%
    Prototype NDJSON failure query 8/8 41,631 -13.8% -2.8%

    Input usage includes repeated/cached context; provider cost is the reported nano-aiu metric, not a USD or energy estimate. The primary run reported 2,170,849 input tokens, including 1,534,979 cached-read tokens. Raw bytes, input-token totals and billed/provider cost are different quantities.

    One footer-first trial had two recovered read-path mistakes, retained in the totals. The two-case real-output calibration did not reproduce a consistent saving: footer-first cost was -0.2%, while CTRF-query was +7.9%, with two recovered read-path mistakes in one query trial. Neither that increase nor the small synthetic reductions establish an intrinsic format effect. A preliminary Windows session-filesystem/tool defect was isolated; its incomplete attempt was excluded from comparisons, not selectively merged into successful trials.

    More actionable findings

    1. Rebaseline the RFC against today's code. Agent-aware output defaults and experimental CTRF JSON execution reporting already exist. The remaining NDJSON question is a stable public incremental contract, not the absence of any execution JSON.
    2. Preserve diagnostics before shortening them. In the real exception-only producer case, the terminal retained an inner diagnostic that CTRF omitted. The synthetic CTRF fixture was more informative than this real report. Full terminal fallback remains necessary; details are relevant to Add JSON / NDJSON test report extension (Microsoft.Testing.Extensions.JsonReport) #8828.
    3. Measure producer cost as well as model consumption. CTRF-on added about 0.49-0.51 seconds to the short controlled host invocations. This includes reporting/recovery/hosting lifecycle, not isolated JSON serialization. Its cause was not profiled.
    4. Separate format from retrieval policy. An appended footer increases emitted output. It helps context only if the tool/agent reads less or avoids another lookup. Selecting failures before giving structured data to the model matters more than the JSON versus text label.

    My recommendation is to keep current defaults, reuse targeted queries over existing reports when those reports are already needed, and prioritize diagnostic fidelity and cheap retrieval. Test #8827 with genuinely large/clipped logs before considering a default. Justify #8828 separately through live-consumer/time-to-first-action requirements.

    Evidence limits

    This is one model, one repeat, eight synthetic cases and two controlled real-output calibrations. It measures exact triage facts, not code-repair success, all agent hosts, other operating systems, live streaming, or servicing branches. There is no measured energy/carbon result. The harness, CSV and raw trajectories are currently retained locally, not committed or hosted as a public benchmark artifact; the figures above are bounded pilot observations rather than an independently downloadable benchmark.

    Primary run: ec9c14a6-002b-4a7c-8598-681685d6618c. Real calibration: 744863c9-d6e4-4d75-b63e-8c2d8af419dd. No shipping implementation or issue-state change accompanied the evaluation.

    Posted by GitHub Copilot at the maintainer's request.

  4. Evangelink commented on Nov 9, 2026

    @Evangelink
    MemberAuthor

    Closing this RFC as completed as an investigation and decision record, not as a claim that every proposed feature shipped. The current checklist combines delivered work, declined designs, and unmeasured hypotheses; the remaining actionable work is now tracked separately.

    Decisions and current state

    Proposal Disposition
    NO_COLOR, agent banner/output defaults, help text and dump noise Completed/merged: #8825, #8771, #8772, #7045.
    Rendering-layer stack-frame filter Not planned: #8826. Declarative StackTraceHidden addressed known platform plumbing; no new rendering deny-list.
    Additional JsonReport/NDJSON package Not planned: #8828. Existing reports, OTel and server mode cover their respective use cases; a separate public stream needs a concrete unmet consumer.
    OTel filtering/privacy #8411 is completed. Existing opt-out/size controls and documentation do not imply automatic redaction of identities, paths or metadata.
    Hidden legacy dotnet test failure details #5188 is completed through documentation. The legacy MSBuild capture default remains; native MTP mode is a different path.
    Failure-index footer #8827 remains open as a clipped-log retrieval evaluation, not a committed default change.
    Heuristic failure deduplication Do not suppress distinct causes or data-row identities using first-line/top-frame grouping. Prefer non-lossy post-processing if a real consumer needs grouping.
    JSON help/info and finer diagnostic codes Conditional capability-discovery/reliability ideas, not justified solely by output size. No implementation commitment without a concrete consumer.

    Focused remaining work

    I am working on the first two follow-ups. They remain open until their actual changes and validation are ready; closing this umbrella does not mark them implemented.

    Evidence and compatibility

    The independent Claude Opus 5.5, Gemini 4 Argon and GPT-6.1 Sol reviews agreed against implementing the checklist wholesale. These are source/design assessments, not a new runtime benchmark. The earlier single-model pilot (#8824 (comment)) had all 54 exact triage grades pass, but small/mixed provider-cost changes and no clipped-log, repair, live-streaming or energy measurement. Output bytes, input tokens, provider cost and energy are different quantities; the earlier automated energy estimates are not measured evidence.

    Shipped MTP 2.5.2 defaults captured stdout/stderr to Failed in detected agent environments and All otherwise. The universal Failed default, non-cursor summary reordering and CTRF graduation are staged under unreleased 3.0 in the changelog, not already shipped 2.x behavior. Progress is independently controlled. New defaults still need explicit compatibility and consumer evidence, even when gated on agent detection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/mtpMicrosoft.Testing.Platform core library.area/terminal-reporterConsole / terminal test reporter.type/discussionOpen discussion / brainstorming.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions