Skip to content
This repository was archived by the owner on Oct 16, 2026. It is now read-only.
This repository was archived by the owner on Oct 16, 2026. It is now read-only.

feat: AgentV Studio — local explorer hardening #563

Closed
Closed
feat: AgentV Studio — local explorer hardening#563
Labels
wuiRelates to the browser dashboard / web UI runtime

Description

@christso
christso
opened on Mar 14, 2026

Status Update (2026-04-08)

AgentV Studio is already implemented as a local browser for eval results via:

  • apps/studio/ (React + Vite)
  • Hono-backed agentv serve / agentv studio
  • run, eval, experiment, target, project, and settings views

This issue should no longer be treated as a greenfield platform build.

Revised Objective

Use #563 for hardening and light polish of the existing Studio explorer, not for building a full eval management platform.

In Scope

  • Small UX improvements to the current Studio explorer
  • Tightening run/project browsing flows
  • Lightweight reuse of existing data primitives where clearly useful
  • Optional lightweight trend/regression surfacing if it cleanly reuses existing CLI/data capabilities

Out of Scope / De-scoped

  • Separate history repo architecture
  • Studio as the primary home for quality gate management
  • Studio as an orchestration command center
  • Building a deep trace-analysis platform inside Studio
  • Replacing external observability tools
  • Public leaderboard / submission workflow
  • Trajectory publishing / shareable links

Why scope changed

  • Observability: AgentV already exports OpenTelemetry to OTLP backends including Langfuse, which covers much of the heavier trace-analysis use case.
  • Quality enforcement: the current workflow is primarily CLI + GitHub Actions CI, not dashboard-driven quality gate management.
  • Platform status: the React/Vite/Hono Studio foundation already exists.

UX Gaps (from SWE-bench competitive analysis)

Competitive analysis of the SWE-bench leaderboard identified specific explorer gaps. See agentevals-research#55 for full analysis.

P1 — High impact, reuses existing CLI data primitives

  • Run comparison view — Select 2 runs, see side-by-side per-test scores with win/loss/tie deltas. Visual equivalent of agentv compare (data already computed by CLI). SWE-bench has a "Compare results" button with checkboxes on every leaderboard tab.
  • Cost column in runs list — Surface total/avg cost per run in the home Runs table. Cost data already exists in JSONL results (cost_usd) but is only visible after drilling into individual eval detail. SWE-bench prominently shows "Avg. $" as a column on every tab.

P2 — Polish, improves explorer usability

  • Column sorting — Click any column header (score, cost, duration, date) to sort. No columns in Studio are currently sortable.
  • Trend visualization — Show agentv trend data as a line chart (score over time by target/suite). CLI already computes linear regression and degrading/improving/stable classification — Studio could render this visually. This would be an AgentV differentiator (SWE-bench has no trend feature).
  • Run filtering — Filter runs list by target, experiment, date range. SWE-bench has agent type and model type dropdowns. Studio currently has tab-based grouping (runs/experiments/targets) but no within-tab filtering.

Implementation notes

All P1 items reuse existing data primitives — no new data collection needed. The run comparison view can call the same logic as agentv compare. Cost is already in JSONL results. This aligns with #788's guidance to keep Studio as a lightweight local explorer that reuses CLI/data primitives.

Remaining Work

  • Audit current Studio UX for obvious explorer gaps (done — SWE-bench competitive analysis, 2026-04-08)
  • Implement P1 gaps (run comparison view, cost column)
  • Implement P2 gaps (column sorting, trend chart, run filtering)
  • Decide whether any additional lightweight trend/regression surface is worth adding
  • Keep docs aligned with Studio's role as a local explorer

Activity

  1. christso commented on Mar 15, 2026

    @christso
    christso
    on Mar 15, 2026
    CollaboratorAuthor

    Status Clarification

    The eval-viewer skill added in plugins/agentv-dev/skills/agentv-bench/eval-viewer/ does not address this issue. It is a single-run review viewer (ported from skill-creator) that renders results.jsonl + optional grading.json into a static HTML page with feedback save/load.

    What eval-viewer covers

    • Reads results.jsonl from a workspace directory
    • Optionally reads grading.json and benchmark.json
    • Serves a self-contained HTML report via a zero-dep Python HTTP server
    • Supports static HTML export (--static)

    What remains unaddressed from this issue

    • History repo architecture (runs/, index.json, history.json, meta.json)
    • .agentv/config.yaml history.repo integration
    • agentv eval run auto-saving snapshots to history repo
    • Progressive SSE visualization (real-time updates during eval runs)
    • Historical trend charts with campaign grouping
    • Trace explorer with collapsible tree from trace.jsonl
    • Cross-run and cross-campaign comparison
    • Dataset browser with filtering and import
    • agentv serve / agentv results reindex / agentv results prune CLI commands
    • Campaign/iteration tracking via history.json

    Of the 15 acceptance signals defined here, 1 is met, 2 are partially met, and 12 are unaddressed. This issue remains fully open.

  2. christso commented on Mar 27, 2026

    @christso
    christso
    on Mar 27, 2026
    CollaboratorAuthor

    Dashboard Scope Upgrade: Viewer → AgentV Studio

    Based on research into melagiri/code-insights and analysis of the current agentv issue landscape, this issue should be upgraded from a read-only viewer to AgentV Studio — a full eval management platform.

    What changes

    Tech stack upgrade:

    • Current plan: Vanilla JS, CSS bar charts, zero dependencies
    • Proposed: React 19 + Vite SPA, Recharts/Plotly for charts, TanStack Table for data grids, shadcn/ui + Tailwind for components
    • Justification: The capabilities below (quality gate management, orchestration monitoring, pattern synthesis) cannot be built with vanilla JS. Code-insights demonstrates this stack works well for a similar use case (React + Vite + Hono).

    New management capabilities (beyond viewing):

    Architecture:

    agentv/
├─ apps/cli/              # Lightweight CLI (unchanged, keeps html-writer.ts)
├─ apps/studio/           # NEW: React SPA (AgentV Studio)
│  ├─ src/pages/
│  │  ├─ Overview.tsx     # Runs overview + stat cards
│  │  ├─ Trends.tsx       # Historical trend charts
│  │  ├─ Gates.tsx        # Quality gate configuration (#334)
│  │  ├─ Alerts.tsx       # Regression alerts (#335)
│  │  ├─ Campaigns.tsx    # Orchestration monitor (#785)
│  │  ├─ RootCause.tsx    # Root cause explorer (#786)
│  │  ├─ Insights.tsx     # Pattern synthesis (#787)
│  │  └─ TraceExplorer.tsx
│  └─ vite.config.ts
└─ apps/server/           # Hono backend (extends existing agentv serve)

    What stays the same:

    • History repo storage model (filesystem, git-based)
    • agentv serve CLI command as entry point (now launches AgentV Studio)
    • Progressive SSE visualization during runs
    • All existing views from the original spec (trends, run history, trace explorer, dataset browser, comparison)
    • html-writer.ts remains for lightweight --format html CI reports

    Tracking issue

    Related new issues

    Research source

    • melagiri/code-insights — React + Vite + Hono dashboard with real-time session analysis, activity charts, cost tracking, pattern detection, dark/light themes
    • Supabase Studio — open-source local-first management UI pattern
  3. christso
    added this to AgentV OSS Boardon Mar 27, 2026
  4. christso
    moved this to Backlog in AgentV OSS Boardon Mar 27, 2026
  5. christso
    moved this from Backlog to Ready in AgentV OSS Boardon Mar 27, 2026
  6. christso
    changed the title [-]feat: self-hosted dashboard — historical trends, dataset management, YAML editor[/-] [+]feat: AgentV Studio — eval management platform with historical trends, quality gates, and orchestration[/+] on Mar 27, 2026
  7. christso
    added
    wuiRelates to the browser dashboard / web UI runtime
    on Mar 27, 2026
  8. christso
    changed the title [-]feat: AgentV Studio — eval management platform with historical trends, quality gates, and orchestration[/-] [+]feat: AgentV Studio — local explorer hardening[/+] on Apr 5, 2026

  9. christso commented on Apr 7, 2026

    @christso
    christso
    on Apr 7, 2026
    CollaboratorAuthor

    Studio UX gaps identified from SWE-bench competitive analysis

    From agentevals-research#55 — SWE-bench leaderboard UX analysis reveals these lightweight explorer gaps in Studio:

    P1 — High impact, aligns with existing CLI primitives

    1. Run comparison view — Select 2 runs, see side-by-side per-test scores with win/loss/tie deltas. This is the visual equivalent of agentv compare, which already produces the data. SWE-bench has a "Compare results" button with checkboxes on the leaderboard.

    2. Cost column in runs list — Surface total/avg cost per run in the home Runs table. Cost data already exists in JSONL results (cost_usd) but isn't shown in the runs list — only visible after drilling into individual eval detail. SWE-bench prominently shows "Avg. $" as a column on every leaderboard tab.

    P2 — Polish, improves explorer usability

    1. Column sorting — Click any column header (score, cost, duration, date) to sort. SWE-bench's % Resolved column is clickable for sorting. Currently no columns in Studio are sortable.

    2. Trend visualization — Show agentv trend data as a line chart (score over time by target/suite). CLI already computes linear regression and degrading/improving/stable classification — Studio could render this visually. SWE-bench doesn't have this (it's leaderboard-only), so this would be an AgentV differentiator.

    3. Run filtering — Filter runs list by target, experiment, date range. SWE-bench has agent type and model type (open source/proprietary) dropdowns. Studio currently has tab-based grouping (runs/experiments/targets) but no within-tab filtering.

    Out of scope (per #788 revised scope)

    • Public leaderboard / submission workflow — not a Studio feature
    • Trajectory publishing / shareable links — requires infrastructure beyond local explorer
    • Benchmark variant tabs — this is dataset management, not Studio UX

    Implementation note

    All P1 items reuse existing data primitives — no new data collection needed. The run comparison view can call the same logic as agentv compare. Cost is already in JSONL results. This aligns with #788's guidance to keep Studio as a lightweight local explorer that reuses CLI/data primitives.

  10. christso
    pinned this issue on Apr 8, 2026

  11. christso commented on Jun 21, 2026

    @christso
    christso
    on Jun 21, 2026
    CollaboratorAuthor

    Thanks, this older Dashboard tracking slice is now folded into the public Dashboard results inspection and scale epic: #1466. Related trace-dashboard work remains under #1451 where applicable. Closing this issue as consolidated so public updates stay in one place.

  12. christso
    closed this as completedon Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wuiRelates to the browser dashboard / web UI runtime

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions