This repository was archived by the owner on Oct 16, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 16, 2026. It is now read-only.
feat(eval): iteration tracking, termination taxonomy, and cross-run regression detection #335
Description
Status Update (2026-04-09)
This issue remains relevant, but its scope should continue to center on core run data and CLI analysis, not a larger assistant-runtime or Studio expansion.
Important note: AgentV already has agentv trend. That means part of the original "cross-run regression detection" goal already exists. The remaining core work is mainly around iteration metadata, termination taxonomy, and artifact-backed run history that other optimization workflows can build on.
Revised Scope
In scope
expected_iterationson testscompletion_signalsupport where appropriatetermination_reason/ related loop metadata in resultsiteration_efficiencyassertion type (still depends on#320)- strengthening/reusing CLI regression analysis primitives where needed
- additive, non-breaking metadata that helps optimization workflows resume and inspect prior cycles
- artifact-backed iteration history primitives where they clearly support CLI analysis and other workflow layers
Deprioritized
- Studio regression alert feed
- Studio regression timeline as a major workstream
- auto-clustering UI inside Studio
- making this issue dependent on a bigger Studio rollout
- persistent personal memory or session-search infrastructure for the core product
Design Boundary
This issue should provide portable run/result primitives that skills, plugins, wrappers, or future workflow tooling can consume. It should not turn AgentV into a long-lived self-improving assistant runtime.
Dependencies
#320remains the important dependency foriteration_efficiency
Acceptance Signals
- iteration metadata is represented in result data cleanly and non-breakingly
- termination reasons are captured for loop-based workflows
-
iteration_efficiencycan be evaluated once#320is available - any regression UX added later can reuse existing CLI/data primitives rather than invent a new subsystem
- result metadata is sufficient for higher-level optimization loops to resume, inspect, and compare prior cycles without requiring chat-memory features
Related
- feat(bench): autoresearch mode — unattended eval-improve loop with hill-climbing ratchet #748 — autoresearch mode in agentv-bench
- agentv-bench: automate keep/discard decision in iteration loop #958 — automated keep/discard in the current bench loop
- tracking: optimization loop roadmap — bench automation, autoresearch, iteration state #1003 — tracking issue for optimization-loop roadmap coordination
Activity
christso commentedon Mar 27, 2026
Extension: Regression Alert Visualization in Dashboard
This issue currently defines the core engine (iteration tracking, termination taxonomy, cross-run regression detection). The following dashboard surface should be added:
Dashboard capabilities for regression detection
How this connects
Research source
christso commentedon Apr 12, 2026
Closing — partially implemented, remainder is not a core concern.
agentv trendalready provides linear regression, slope thresholds, and--fail-on-degradingCI gatingcode-graderthat reads the trace event countcode-gradercan parse agent output/trace and classify terminationSee #337 for full rationale.