Repository navigation
Open
Labels
area/mtpMicrosoft.Testing.Platform core library.Microsoft.Testing.Platform core library.area/terminal-reporterConsole / terminal test reporter.Console / terminal test reporter.
Description
Summary
At the end of a Microsoft.Testing.Platform test run, emit a compact, sentinel-delimited footer that lists every failed test as a single line of Test FQN | first-line error | file:line. Default-on in LLM-detected environments only (see LLMEnvironmentDetector).
Motivation
When a test run fails, an LLM agent currently has to re-read the entire terminal transcript to figure out what failed and where. The per-failure blocks are paragraph-style (message + expected/actual + stack), suitable for a human scanning visually but expensive in tokens when the agent only needs the structured summary to plan its next step.
A small machine-friendly footer at a stable position is the lowest-cost way to give the agent a "table of contents" of failures.
Proposed shape
##[failures]
TestFx.UnitTests.FooTests.WhenBarThenBaz | Expected 42 but was 41 | src/Foo.cs:128
TestFx.UnitTests.FooTests.WhenBarThenQux | NullReferenceException: Object reference not set... | src/Foo.cs:142
##[/failures]
Properties:
- Sentinel-delimited (
##[failures]...##[/failures]) so an agent can extract the block with a 2-line regex without parsing the rest of the transcript. - One line per failure. Fields separated by
|. Pipe-in-value escaped as\|. - Test FQN comes from the test node display path.
- First-line error: first non-empty line of the failure message, truncated to ~200 chars (configurable).
file:lineextracted from the first user-code frame (after the filter from Honor NO_COLOR environment variable in MTP terminal output #8825 / related). Empty if not found.
Behavior
- Default-on when
LLMEnvironmentDetector.IsLLMEnvironment()returnstrue. - Default-off otherwise, to keep the human terminal transcript stable.
- For v1 do not add a CLI option, to avoid
--help/--infoacceptance churn and the XLF round. - If desired in v2, add
--failure-summary <auto|on|off>.
Implementation notes
TerminalTestReporteralready tracks per-assembly state and emits anAppendTestRunSummaryat the end (seeTerminalTestReporter.Summary.cs). The failure index should be appended right after that summary block.- Today
TerminalTestReporter.TestCompletedrenders failed details and discards them. To build the index, capture a bounded list of(displayName, firstErrorLine, firstUserCodeFileLine)per failed test as we render it. - Cap the captured list at e.g. 5,000 entries to avoid OOM in giant runs; if exceeded, emit
## ...truncated, N more failures.... - Verify behavior under multi-assembly runs (one footer per assembly vs one global footer at the end — global at the end is preferable for agents).
Risks
- Adding any new lines to the terminal transcript can break existing scrapers. Mitigation: LLM-mode-gated default.
- Capturing failure state grows reporter memory. Mitigation: cap + truncate.
Acceptance criteria
- In LLM mode, after a failed run the transcript contains exactly one
##[failures]/##[/failures]block listing all failures. - In human mode (no LLM env vars), output is identical to today.
- Tests cover: zero failures (no block), single failure, many failures, multi-line error message, missing file path.
References
- Umbrella: [RFC] Agent/LLM-efficient test output for Microsoft.Testing.Platform #8824
- Files:
src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.*.cs - Inspiration: GitHub Actions log grouping (
::group::), Azure DevOps##[error]/##vso[task.logissue]lines.
Metadata
Metadata
Assignees
Labels
area/mtpMicrosoft.Testing.Platform core library.Microsoft.Testing.Platform core library.area/terminal-reporterConsole / terminal test reporter.Console / terminal test reporter.
Activity
Evangelink commentedon Oct 15, 2026
Evaluation follow-up: distinguish an additive footer from a retrieval policy
The local output-value pilot for #8824 gives weak evidence for changing the default, rather than proof that a failure index cannot help.
At TestFx revision
6c6b1f474acbb6ab1bdbdec73fa32568f12dfcb5, eight synthetic scenarios were compared with identical prompts, exact graders, run metadata and full-detail fallback. Executor:gpt-6.1-sol, Vally 0.14.0, Copilot SDK 1.0.11 / nested CLI 1.0.80, one worker and one repeat.details.logreferencesThe last column counts tool calls explicitly referencing the detail file, not a guaranteed count of successful reads. One footer-first trial had two recovered path-lookup mistakes; these remain included. Its clean-run input bytes were identical to the terminal arms, so the extra turns cannot be treated as proof that the footer intrinsically costs more.
The separate two-case calibration used actual terminal output from locally packed MTP
2.6.0-dev/ CTRF1.0.0-dev, with a controlled framework on Windows x64/net9.0. Footer-first saved only 0.6% input and 0.2% reported provider cost there, and both cases still referred to the full diagnostic transcript. These samples are too small for a population ranking.What this means for the design
[INVOCATION_FAILED] Invocation failed.omitted the inner[SERVICE_AUTH_401]diagnostic needed by the exact task. Correct handling required drill-down.Recommendation: keep this lower priority or opt-in until a genuinely large/clipped-output experiment demonstrates a practical end-to-end benefit. That experiment should compare current agent defaults, additive footer, tail/index-first with drill-down, and an existing structured-report query using the same outcomes.
This pilot used file-first inputs, not a real
tailpolicy over clipped command output. It does not rule out that use case, and it did not implement a shipping footer. All exact triage answers passed; code repair, live output, repeated reliability and other models were not measured. Cost deltas are provider-reported usage, not USD or energy. Full harness/traces remain local rather than a public downloadable artifact; the broader method and comparison are summarized on #8824.Posted by GitHub Copilot at the maintainer's request.
Evangelink commentedon Nov 9, 2026
The #8824 multi-model review recommends keeping this as a focused retrieval experiment, not enabling an automatic footer now. Report-fidelity work is tracked separately in #12274 and version-aware agent guidance in #12275; neither requires a new footer.
The earlier pilot (#8827 (comment)) did not test genuinely clipped command output or code repair. It does not establish a consistent end-to-end benefit, but it also does not rule out an index for large logs.
Before implementation, compare current agent-aware output, an additive footer, tail/index-first retrieval with full drill-down, and a failure query over an existing report. Use real logs exceeding the actual consuming tool's truncation limit. Measure correct diagnosis/repair, total reads and model turns, and producer overhead—not just the size of the first read. Keep outcomes and fallback evidence identical across arms.
Any index must identify final failures after retries, retain module/TFM and data-row identity, distinguish incomplete/aborted runs from success, and declare bounded omissions. A display name is not necessarily an FQN or rerun key. A first-line summary is an index, not a replacement for full exception details. LLM detection does not make a default output change compatibility-safe, and avoiding help/localization updates is not a design justification for omitting an opt-out.
This issue remains open; no shipping footer or new default is being claimed.