Skip to content

Emit compact end-of-run failure index for LLM/agent consumption #8827

Description

Summary

At the end of a Microsoft.Testing.Platform test run, emit a compact, sentinel-delimited footer that lists every failed test as a single line of Test FQN | first-line error | file:line. Default-on in LLM-detected environments only (see LLMEnvironmentDetector).

Motivation

When a test run fails, an LLM agent currently has to re-read the entire terminal transcript to figure out what failed and where. The per-failure blocks are paragraph-style (message + expected/actual + stack), suitable for a human scanning visually but expensive in tokens when the agent only needs the structured summary to plan its next step.

A small machine-friendly footer at a stable position is the lowest-cost way to give the agent a "table of contents" of failures.

Proposed shape

##[failures]
TestFx.UnitTests.FooTests.WhenBarThenBaz | Expected 42 but was 41 | src/Foo.cs:128
TestFx.UnitTests.FooTests.WhenBarThenQux | NullReferenceException: Object reference not set... | src/Foo.cs:142
##[/failures]

Properties:

  • Sentinel-delimited (##[failures] ... ##[/failures]) so an agent can extract the block with a 2-line regex without parsing the rest of the transcript.
  • One line per failure. Fields separated by |. Pipe-in-value escaped as \|.
  • Test FQN comes from the test node display path.
  • First-line error: first non-empty line of the failure message, truncated to ~200 chars (configurable).
  • file:line extracted from the first user-code frame (after the filter from Honor NO_COLOR environment variable in MTP terminal output #8825 / related). Empty if not found.

Behavior

  • Default-on when LLMEnvironmentDetector.IsLLMEnvironment() returns true.
  • Default-off otherwise, to keep the human terminal transcript stable.
  • For v1 do not add a CLI option, to avoid --help/--info acceptance churn and the XLF round.
  • If desired in v2, add --failure-summary <auto|on|off>.

Implementation notes

  • TerminalTestReporter already tracks per-assembly state and emits an AppendTestRunSummary at the end (see TerminalTestReporter.Summary.cs). The failure index should be appended right after that summary block.
  • Today TerminalTestReporter.TestCompleted renders failed details and discards them. To build the index, capture a bounded list of (displayName, firstErrorLine, firstUserCodeFileLine) per failed test as we render it.
  • Cap the captured list at e.g. 5,000 entries to avoid OOM in giant runs; if exceeded, emit ## ...truncated, N more failures....
  • Verify behavior under multi-assembly runs (one footer per assembly vs one global footer at the end — global at the end is preferable for agents).

Risks

  • Adding any new lines to the terminal transcript can break existing scrapers. Mitigation: LLM-mode-gated default.
  • Capturing failure state grows reporter memory. Mitigation: cap + truncate.

Acceptance criteria

  • In LLM mode, after a failed run the transcript contains exactly one ##[failures]/##[/failures] block listing all failures.
  • In human mode (no LLM env vars), output is identical to today.
  • Tests cover: zero failures (no block), single failure, many failures, multi-line error message, missing file path.

References

Activity

  1. Evangelink commented on Oct 15, 2026

    @Evangelink
    MemberAuthor

    Evaluation follow-up: distinguish an additive footer from a retrieval policy

    The local output-value pilot for #8824 gives weak evidence for changing the default, rather than proof that a failure index cannot help.

    At TestFx revision 6c6b1f474acbb6ab1bdbdec73fa32568f12dfcb5, eight synthetic scenarios were compared with identical prompts, exact graders, run metadata and full-detail fallback. Executor: gpt-6.1-sol, Vally 0.14.0, Copilot SDK 1.0.11 / nested CLI 1.0.80, one worker and one repeat.

    View Correct Mean input tokens Input change Reported provider-cost change Explicit details.log references
    Full terminal baseline 8/8 48,306 baseline baseline 6
    Same terminal + prototype index 8/8 43,357 -10.2% -2.1% 3
    Summary + prototype index first 8/8 51,519 +6.7% +1.3% 6

    The last column counts tool calls explicitly referencing the detail file, not a guaranteed count of successful reads. One footer-first trial had two recovered path-lookup mistakes; these remain included. Its clean-run input bytes were identical to the terminal arms, so the extra turns cannot be treated as proof that the footer intrinsically costs more.

    The separate two-case calibration used actual terminal output from locally packed MTP 2.6.0-dev / CTRF 1.0.0-dev, with a controlled framework on Windows x64/net9.0. Footer-first saved only 0.6% input and 0.2% reported provider cost there, and both cases still referred to the full diagnostic transcript. These samples are too small for a population ranking.

    What this means for the design

    • The footer is an index, not replacement evidence. A first line such as [INVOCATION_FAILED] Invocation failed. omitted the inner [SERVICE_AUTH_401] diagnostic needed by the exact task. Correct handling required drill-down.
    • Do not count smaller first reads as total savings. Appending an index makes the emitted transcript larger. The useful outcome is fewer total tool/model reads at unchanged correctness, including fallback.
    • Record final failures, not every rendered failure. A failed attempt followed by a passing retry must not remain in the final-failure index. The current reporter already reconciles retries and multiple modules; see completion handling and summary aggregation.
    • A display name is not necessarily an FQN or rerun key. Module/TFM and framework UID/data-row identity need a defined scope, especially with duplicate names.
    • Bound bytes/characters as well as entries, and declare omissions. A capped index cannot silently imply that it lists every failure. Preserve process/completion status separately so no block, a crash or a partial run cannot look successful. Escaping must handle backslashes, pipes, newlines and test-produced sentinel lookalikes.

    Recommendation: keep this lower priority or opt-in until a genuinely large/clipped-output experiment demonstrates a practical end-to-end benefit. That experiment should compare current agent defaults, additive footer, tail/index-first with drill-down, and an existing structured-report query using the same outcomes.

    This pilot used file-first inputs, not a real tail policy over clipped command output. It does not rule out that use case, and it did not implement a shipping footer. All exact triage answers passed; code repair, live output, repeated reliability and other models were not measured. Cost deltas are provider-reported usage, not USD or energy. Full harness/traces remain local rather than a public downloadable artifact; the broader method and comparison are summarized on #8824.

    Posted by GitHub Copilot at the maintainer's request.

  2. Evangelink commented on Nov 9, 2026

    @Evangelink
    MemberAuthor

    The #8824 multi-model review recommends keeping this as a focused retrieval experiment, not enabling an automatic footer now. Report-fidelity work is tracked separately in #12274 and version-aware agent guidance in #12275; neither requires a new footer.

    The earlier pilot (#8827 (comment)) did not test genuinely clipped command output or code repair. It does not establish a consistent end-to-end benefit, but it also does not rule out an index for large logs.

    Before implementation, compare current agent-aware output, an additive footer, tail/index-first retrieval with full drill-down, and a failure query over an existing report. Use real logs exceeding the actual consuming tool's truncation limit. Measure correct diagnosis/repair, total reads and model turns, and producer overhead—not just the size of the first read. Keep outcomes and fallback evidence identical across arms.

    Any index must identify final failures after retries, retain module/TFM and data-row identity, distinguish incomplete/aborted runs from success, and declare bounded omissions. A display name is not necessarily an FQN or rerun key. A first-line summary is an index, not a replacement for full exception details. LLM detection does not make a default output change compatibility-safe, and avoiding help/localization updates is not a design justification for omitting an opt-out.

    This issue remains open; no shipping footer or new default is being claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/mtpMicrosoft.Testing.Platform core library.area/terminal-reporterConsole / terminal test reporter.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions