tracking: optimization loop roadmap — bench automation, autoresearch, iteration state #1003

Objective

Coordinate AgentV's optimization-loop roadmap across the existing bench/workflow issues without duplicating implementation scope.

This issue is a parent tracker for the current optimization-loop cluster: small iterative improvements in the bench skill, unattended autoresearch mode, mutation/bootstrap helpers, and the core run metadata needed to support those workflows cleanly.

Design Latitude

Use this issue only to coordinate sequencing, boundaries, and dependencies across the linked issues.

What belongs here

• clarifying implementation order across the linked issues
• keeping the scope split clean between skill/workflow work and core artifact/data work
• tracking when a child issue is now redundant because another issue already covers it
• keeping the roadmap legible as the optimization-loop story evolves

What does not belong here

• new implementation scope that duplicates a child issue
• assistant-runtime features unrelated to AgentV's eval-runner role
• persistent personal memory, messaging continuity, or self-improving general-agent product work

Current issue map

Recommended implementation order

Phase 1: immediate workflow improvements

1. agentv-bench: automate keep/discard decision in iteration loop #958 — automate keep/discard in the current bench loop
2. feat(bench): mutator subagent — autonomous artifact rewriting from failure analysis #746 — mutator subagent
3. feat(eval-writer): eval-generator subagent — bootstrap EVAL.yaml from existing artifacts #747 — eval-generator bootstrap

Phase 2: unattended loop

4. feat(bench): autoresearch mode — unattended eval-improve loop with hill-climbing ratchet #748 — full autoresearch mode built on the narrower workflow pieces

Phase 3: core data hardening

5. feat(eval): iteration tracking, termination taxonomy, and cross-run regression detection #335 — enrich iteration metadata and artifact-backed loop history where the higher-level workflow clearly needs stable core primitives

Acceptance Signals

• the linked issues have clear non-overlapping scopes
• the bench-skill improvements stay separate from core metadata work
• the full autoresearch loop remains artifacted and resumable, not runtime-memory-driven
• no child issue grows into a self-improving assistant-runtime project outside AgentV's product boundary

Non-Goals

• Not a replacement for the child issue bodies as the implementation contracts
• Not a request to build all optimization-loop ideas in the core product
• Not a mandate to add personal memory, session search, or runtime-managed skill storage to AgentV

Related

• agentv-bench: automate keep/discard decision in iteration loop #958
• feat(bench): mutator subagent — autonomous artifact rewriting from failure analysis #746
• feat(eval-writer): eval-generator subagent — bootstrap EVAL.yaml from existing artifacts #747
• feat(bench): autoresearch mode — unattended eval-improve loop with hill-climbing ratchet #748
• feat(eval): iteration tracking, termination taxonomy, and cross-run regression detection #335
• feat: Pass@k Trial Strategy for LLM Non-Determinism #214