feat: remove workspace_template from targets, add target-level hooks for harness variant testing #1094

Summary

Remove the unused workspace_template field from the target schema and add target-level hooks support, so a single eval file can compare different harness setups (e.g., baseline vs with-guidelines) by running different setup scripts per target in the eval-owned workspace.

Motivating example: #1091

PR #1091 (bug-fix-benchmark showcase) demonstrates the problem. It compares 4 plugin configurations (baseline, superpowers, compound, agent-skills) and requires:

• 4 near-identical eval files (bug-fixes-baseline.eval.yaml, bug-fixes-superpowers.eval.yaml, etc.) — same test cases, same assertions, only workspace differs
• 4 workspace template directories — each with a different settings.json
• A setup-plugins.sh script to install plugins into each workspace

With target-level hooks, this collapses to 1 eval file:

# evals/bug-fixes.eval.yaml
workspace: ./workspaces/baseline/

targets:
  - name: claude-baseline

  - name: claude-superpowers
    use_target: claude
    hooks:
      before_each:
        command: setup-plugins.sh superpowers

  - name: claude-compound
    use_target: claude
    hooks:
      before_each:
        command: setup-plugins.sh compound

  - name: claude-agent-skills
    use_target: claude
    hooks:
      before_each:
        command: setup-plugins.sh agent-skills

No test case duplication. No drift risk. One agentv eval run compares all variants.

Motivation

When comparing workspace variants, the current design requires separate eval files that duplicate test cases and assertions. This causes test case drift, conceptual muddling, and scaling pain (N variants = N files).

Industry best practice is clear — the test specification owns the environment, the thing being tested doesn't:

• pytest: fixtures own the environment, parametrize varies config
• GitHub Actions: job defines the environment, matrix varies parameters, steps are scripts
• SWE-bench: harness sets up the repo/environment, agent submission is just the agent
• Inspect AI: task defines dataset + sandbox, solver is just the agent logic

AgentV's targets.yaml is already clean — pure provider/model/auth. Workspace config belongs in the eval, not the target.

Design rationale

Why workspace_template was on targets and why it should be removed

The original design allowed workspace_template: string on targets, letting a target specify a template directory for its workspace. This conflated two concerns: "who runs the task" (provider/model/auth) with "what environment the task runs in" (workspace). In practice, nobody uses it — zero references in WTG.AI.Prompts (113 eval files), zero references in any agentv eval file. All workspace config is specified at the eval level, which is correct.

Why we're NOT putting workspace config back on targets

During design discussion, we considered three approaches for harness A/B comparison:

1. Full WorkspaceConfig on targets (target overrides eval workspace) — Rejected. This re-introduces the same conflation. It also raises unresolvable merge vs override questions: if both eval and target define repos, hooks, and mode, what wins? Override means targets must duplicate all eval-level setup. Merge requires per-field merge strategies and becomes hard to reason about.

2. extends: eval composition (target workspace layers on top of eval workspace) — Rejected. Merge semantics for repos (concat?), hooks (chain?), mode (override?) add complexity with no clear benefit. Every new workspace field would need a merge strategy defined.

3. Target-level hooks only (target runs scripts in the eval's workspace) — Chosen. The eval owns the workspace. The target optionally runs a setup script to customize it for its variant. No merge semantics. No new concepts. Scripts are procedural and explicit — they do exactly what they say.

Core architectural principle

targets.yaml  = WHO runs the task    (provider, model, auth, grader)
eval.yaml     = WHAT to test + WHERE (tests, assertions, workspace)
target hooks  = HOW to customize     (scripts that configure the workspace per variant)

Targets never define or own workspace configuration. The eval file is the single source of truth for the environment. This matches industry standard practice and keeps each layer single-purpose.

Changes

1. Remove workspace_template from target schema

workspace_template: z.string().optional() on BASE_TARGET_SCHEMA is unused by any downstream consumer (verified: zero usage in WTG.AI.Prompts, zero usage in any eval file). Remove it entirely — no deprecation period needed.

Files requiring changes (14 references):

• packages/core/src/evaluation/providers/targets.ts — schema definition
• packages/core/src/evaluation/providers/types.ts — TypeScript interface
• packages/core/src/evaluation/types.ts — type references
• packages/core/src/evaluation/orchestrator.ts — runtime usage
• packages/core/src/evaluation/evaluators/types.ts — evaluator types
• packages/core/src/evaluation/evaluators/llm-grader.ts — grader usage
• packages/core/src/evaluation/providers/vscode-provider.ts — VS Code provider
• packages/core/src/evaluation/providers/codex-cli.ts — Codex provider
• packages/core/src/evaluation/validation/targets-validator.ts — validator
• packages/core/test/evaluation/providers/targets.test.ts — tests
• packages/core/test/evaluation/validation/targets-validator.test.ts — tests
• apps/web/src/content/docs/docs/targets/configuration.mdx — docs
• apps/web/src/content/docs/docs/targets/coding-agents.mdx — docs
• plugins/agentv-dev/skills/agentv-eval-writer/SKILL.md — skill docs

2. Add target-level hooks for per-target workspace customization

Allow targets (as referenced in eval files) to run setup/teardown scripts in the eval's workspace. This enables harness variant comparison without duplicating eval files.

# eval.yaml — eval owns the workspace
workspace: ./workspace.yaml

targets:
  - name: baseline
    # inherits workspace as-is, no hooks

  - name: with-skills
    use_target: default
    hooks:
      before_each:
        command: copy-skills.mjs --plugin cargowise

  - name: with-skills-and-guidelines
    use_target: default
    hooks:
      before_each:
        command: |
          copy-skills.mjs --plugin cargowise
          cp guidelines.md $WORKSPACE_PATH/.claude/

Hook execution order

For a target with hooks, the full lifecycle is:

1. Workspace materializes (repos cloned, template copied)
2. Workspace before_all hook runs
3. Target before_all hook runs
4. For each test:
   a. Workspace before_each hook runs
   b. Target before_each hook runs
   c. Test executes
   d. Target after_each hook runs
   e. Workspace after_each hook runs
5. Target after_all hook runs
6. Workspace after_all hook runs

Target hooks run after workspace hooks on setup, before workspace hooks on teardown (standard setup/teardown nesting).

Schema

Target hooks follow the same schema as workspace hooks:

hooks?: {
  before_all?: { command: string; timeout?: number };
  before_each?: { command: string; timeout?: number };
  after_each?: { command: string; timeout?: number };
  after_all?: { command: string; timeout?: number };
}

Where target hooks are defined

Target hooks are defined in the eval file, not in targets.yaml. This is intentional — the hooks are eval-specific customization, not part of the target's identity:

# targets.yaml — clean, no workspace concerns
targets:
  - name: default
    use_target: ${{ AGENT_TARGET }}
  - name: claude
    provider: claude-cli
    grader_target: grader

# eval.yaml — hooks are eval-specific
targets:
  - name: baseline           # references "default" from targets.yaml
  - name: with-skills        # same target, different setup
    use_target: default
    hooks:
      before_each:
        command: copy-skills.mjs --plugin cargowise

Use cases

1. Harness optimization — same model, same tests, different setup scripts per target (skills, guidelines, tools)
2. A/B testing prompts — same workspace, same model, different prompt/guidelines copied in via hooks
3. Plugin comparison — same bug fix tasks, different plugins installed per target (see docs(showcase): add bug-fix-benchmark example for SWE-bench style evaluation #1091)
4. Model comparison — different targets, same workspace, no hooks needed (already works)
5. Full matrix — different targets with different hooks in one eval file

Downstream impact (WTG.AI.Prompts)

• workspace_template on targets: zero usage — safe to remove
• Eval-level workspace references: unaffected
• Target hooks: opt-in — available when they want to compare harness variants without duplicating eval files
• Current pattern of copy-local-skills.mjs in workspace hooks maps directly to target hooks

Implementation notes

• Target hooks should have access to the same environment variables as workspace hooks (e.g., $WORKSPACE_PATH)
• When a target has no hooks, behavior is identical to today — no changes for existing evals
• The workspace_template removal and target hooks addition can be done as separate PRs if preferred, since removal is a breaking schema change (even though unused)
• PR docs(showcase): add bug-fix-benchmark example for SWE-bench style evaluation #1091 can be updated as a showcase of the new pattern once target hooks are implemented