This repository was archived by the owner on Oct 16, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 16, 2026. It is now read-only.
feat: snapshot MCP server pattern for reproducible benchmarks #1135
Description
Objective
Document and provide tooling for including MCP servers in eval benchmarks without introducing look-ahead bias or non-reproducibility.
Problem
Real-world agent harnesses include MCP servers (filesystem, database, API integrations). But including live MCP servers in benchmarks creates two problems:
- Look-ahead bias — live data at eval time may contain information unavailable when the task was authored. The agent could find the answer in post-dated data.
- Non-reproducibility — live data changes between runs, so the same eval produces different results on different days. Regression detection becomes unreliable.
Stripping MCP servers from benchmarks avoids these problems but means you're not testing the real harness — you're testing a hobbled version of the agent.
Proposed approach: snapshot MCP
Freeze MCP server data at task-authoring time:
- Live database MCP → snapshot the database, serve it read-only
- Live API MCP → record API responses, replay them (VCR/cassette pattern)
- Live search MCP → pre-cache search results for the task's domain
The MCP server itself is real (the agent calls real MCP tools), but the data it serves is frozen.
Integration with AgentV's workspace model
# EVAL.yaml
workspace: ./cases/customer-lookup/
setup:
- command: ./start-snapshot-mcp.sh # starts MCP server with frozen data
teardown:
- command: ./stop-snapshot-mcp.sh
tests:
- id: customer-lookup
input: "Find the customer who placed the most orders last quarter"
execution:
evaluators:
- type: tool-trajectory
expected_tools:
- name: mcp__testdb__query
- type: code-grader
command: ./verify-answer.shDeliverables
- Documentation: Guide on snapshot MCP patterns for benchmarks (workspace setup, data freezing strategies)
- Example: At least one example eval with a snapshot MCP server (e.g., SQLite database served via MCP)
- Tooling (stretch): Helper script or utility for recording/replaying MCP server responses
Decision matrix for users
| Approach | Look-ahead bias | Reproducibility | Harness fidelity | Best for |
|---|---|---|---|---|
| No MCP in eval | None | Perfect | Low | Unit-style skill tests |
| Snapshot MCP (frozen data) | None | High | High | Publishable benchmarks |
| Synthetic/hermetic MCP | None | Perfect | Medium | Cross-agent comparison |
| Time-bounded live MCP | Possible | Moderate | Highest | Daily regression tracking |
Related issues
- bug: copilot CLI MCP auth prompt blocks eval execution #461 — MCP auth prompt blocks eval execution (related: MCP servers in eval context)
Non-goals
- Not building a generic MCP recording framework — just patterns and lightweight tooling
- Not requiring all evals to use snapshot MCP — this is an opt-in pattern
Acceptance signals
- Documentation guide published
- At least one working example with snapshot MCP
- Pattern is reproducible: same eval produces same results across runs
Activity
christso commentedon Apr 17, 2026
Closing — this is already achievable with existing workspace primitives.
A snapshot MCP server for benchmarks is just workspace setup:
before_allhook → starts the MCP serverafter_allhook → stops itNo new tooling needed. If there's user demand for a convenience helper (e.g., "record MCP responses and replay them"), that can be a separate issue, but the core pattern works today.