This repository was archived by the owner on Oct 16, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 16, 2026. It is now read-only.
feat(eval): composable quality gates with auto-remediation triggers #334
Description
Status Update (2026-04-05)
This issue remains valid as a core evaluation/CLI capability issue.
The earlier Studio expansion is no longer the default plan. Quality enforcement today is primarily handled by AgentV eval thresholds + GitHub Actions CI, so the dashboard-management portion of this issue should be treated as optional follow-up work, not part of the critical path.
Revised Scope
In scope
severity: error | warning | infoon evaluator configs- non-blocking warnings / informational outcomes in result data
- optional remediation hooks if they are clearly useful in CLI/automation workflows
- reusable gate patterns via code graders / scripts / packages
Deprioritized
- Dashboard-driven gate CRUD
- Visual threshold editor in Studio
- Alert routing / remediation center in Studio
- Gate compliance dashboards in Studio
Notes
If Studio ever needs a UI for this later, it should come after the core primitives exist and only if CI + CLI workflows prove insufficient.
Acceptance Signals
-
severity: warning|error|infoworks on evaluator configs - warnings are visible in output/result data without failing the eval
- errors continue to block the eval as today
- any remediation hook design is usable from CLI/automation flows
Activity
christso commentedon Mar 27, 2026
Extension: Dashboard-Driven Gate Configuration
This issue currently defines the core engine (severity levels, auto-remediation triggers, reusable gate library). The following dashboard surface should be added as part of the AgentV Studio (#563 upgrade):
Dashboard capabilities for quality gates
How this connects
Research source
christso commentedon Apr 12, 2026
Closing as YAGNI — the existing
required+weightfields already express severity semantics:required: true= error (must pass)required: false(default) = warning (contributes to score)weight: 0= info (tracked but doesn't affect score)Auto-remediation hooks belong in CI/CD pipelines, not the eval framework. See #337 for full rationale.