Repository navigation
fix(eval): pin the D1 dataset's upstream servers, all seven - #931
Merged
Merged
Conversation
…hot config
The "Retrieval regression gate (D1)" job fails because the eval dataset config
declares its 7 reference MCP servers with no version pin, so npx/uvx resolve
"latest" at CI time.
On 2026-07-28 the MCP Python SDK published 2.0.0, which renamed
`McpError` -> `MCPError` and dropped the low-level `Server.list_tools` /
`Server.list_resources` decorators. All four uvx servers declare only a lower
bound (fetch `mcp>=1.1.3`, git `mcp>=1.0.0`, time `mcp>=1.23.0`,
sqlite `mcp[cli]>=1.6.0`), so every one of them resolved 2.0.0 and crashed on
import/startup — not just `fetch`:
fetch, time -> ImportError: cannot import name 'McpError' from
mcp.shared.exceptions ... Did you mean: 'MCPError'?
git, sqlite -> AttributeError: 'Server' object has no attribute
'list_tools' / 'list_resources'
Only the three npx servers connected: 24 tools instead of 45, so the gate's
catalog-readiness check (expected>=44) timed out and the job exited 1.
Pin every server to an exact version, and additionally pin the SDK for the uvx
servers via `--with mcp==1.29.0` (the last 1.x). Pinning only the server
packages would not have helped here — the break came from a transitive
dependency, so the constraint has to name it.
An unpinned dependency inside a *regression gate* is a latent flake by
construction: the corpus, golden set and baseline are frozen, so any upstream
publish silently redefines the universe being scored and the gate turns red on
someone else's release schedule. The npx servers were not broken today, but
they carry exactly the same exposure and are pinned for the same reason.
The pinned versions reproduce the frozen 45-tool corpus_v1 tool-for-tool
(filesystem 14, git 12, memory 9, sqlite 6, time 2, fetch 1,
sequential-thinking 1), verified by probing each server directly and by a full
local re-run of the CI gate step: catalog reached 45 tools,
Recall@5 = 0.681 vs threshold 0.631 -> PASS.
Same config also backs the D4 bench job and bench/docker-compose.yml, so both
gates become reproducible. README documents the pinning contract and that
bumping a pin is a corpus change (cut corpus_v2, never edit corpus_v1).
Deploying mcpproxy-docs with
|
| Latest commit: |
cc44436
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://070835e6.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://fix-d1-gate-pin-eval-upstrea.mcpproxy-docs.pages.dev |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 30516508861 --repo smart-mcp-proxy/mcpproxy-go
|
…es exactly The previous pins froze 'latest at pin time' (July 2026), not the versions current at the corpus freeze (2026-05-31): server-filesystem@2026.7.10 drops read_file (44-tool catalog, not 45) and server-sequential-thinking@2026.7.4 renames sequentialthinking -> sequential_thinking, so two golden queries could never score and the gate passed only on threshold slack. Repin all seven servers + the MCP Python SDK to freeze-era releases, verified tool-for-tool against a live /api/v1/tools snapshot: 45/45 canonical ids match corpus_v1 exactly. Tighten the eval.yml readiness check from >=44 to the full 45 now that the catalog is deterministic, and document what the pins do and do not freeze (immutable versions pin the tool list; transitive deps still resolve at install time and can only fail loudly at startup, not drift silently).
The readiness loop only counted tools (>=45), so a same-count rename or substitution — exactly the drift class that broke the golden set — would still pass readiness and silently skew recall. After the catalog settles, compare the live canonical server:tool ID set against corpus_v1.tools.json and fail the job with the precise missing/extra diff if they differ.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Dumbris
added a commit
that referenced
this pull request
Aug 31, 2026
… round 5) The final review round caught an error in this PR's own flip, and it is the same failure mode the PR was written to fix: an epic marked done while a governing FR is unmet. Spec 065 FR-009 and SC-005 require CI to FAIL on a discovery regression beyond tolerance. The retrieval-D1 job sets continue-on-error for pull_request (D1 depends on npx/uvx fetches, a known flake source), so on the PR path CI does not fail — and eval.yml's own comment records the promotion to PR-blocking as still planned (MCP-742). The HARNESS shipped; the GATE did not. Reverted to in_progress and added the remaining work as a child task (discovery-eval-pr-blocking, MCP-742) so the epic's status has something in the DAG to point at. The reviewer also read CN-002 (frozen corpus, never live) as violated because D1 boots a live mcpproxy serving seven reference servers. Recorded as a weaker tension to adjudicate rather than a violation: #931 pinned all seven upstreams to freeze-era versions and the job gates on the exact corpus ID set, so the corpus is reproducible in practice. Whether that satisfies CN-002 or a committed snapshot is required is a judgement for the spec owner. gen-roadmap.py --check and --check-github both clean (0 errors, 0 warnings).
Dumbris
added a commit
that referenced
this pull request
Aug 31, 2026
…ithub warnings (#1136) * chore(roadmap): truth-sync statuses against code, clear all --check-github warnings An evidence-based re-derivation of every non-done roadmap.yaml leaf. Each flip below was verified against the tree, git/gh state and the rendering sites, then independently re-checked by an adversarial pass; claims that did not survive that pass were dropped rather than applied. Statuses that were wrong in the "already shipped" direction: - upgrade-nudge-delta todo -> done (#1085, closes #1081). The note asserted TODO(spec-079/FR-002) markers exist and that the checker calls only /releases/latest; both are false. internal/updatecheck/delta.go now computes the delta and all six surfaces render it. - upgrade-nudge epic in_progress -> done (all five tasks now done). - analytics-dashboard in_progress -> done. Both children shipped; the default-landing half is live (router '/' -> Dashboard, guarded by dashboard-default-landing.spec.ts). Spec 069's one open task is a local Playwright sweep that leaves no artifact and can never tick from code. - discovery-eval-harness todo -> done. Both halves ship: cmd/scan-eval for security recall/FP, and the eval.yml retrieval-d1 gate plus bench/'s own RecallAtK/NDCGAtK + recall@5 parity gate for discovery quality. - hygiene-spec-gardener in_progress -> done (#824, #870, #999, #842). Its recorded blocker is cleared: check-spec-evidence.py is on main. Statuses that were wrong in the "claimed progress it does not have" direction: - windows-tray and windows-tray-window in_review -> todo. No Windows-tray PR is open or merged and native/windows/ is a README placeholder, so "in review" pointed at nothing. The section header is renamed to match. - planning-hygiene todo -> in_progress and action-log-transparency todo -> in_progress; both already have done children. Notes corrected where they would misdirect the next implementer: - token-bench-harness claimed it absorbs discovery-eval-harness by measuring retrieval recall. Recall is already measured, and bench/ already covers token counting, encoding arms (incl. direct_deferred), break-even and reporting. The note now records what is genuinely open: activity-log session replay, measured first-call success/retry rate, and tokens-per-successful-task. - token-bench-public: the MCP-specific third already ships via spec 083 (ToolRet, LiveMCPBench snapshot, lap-score); only agent-loop suites remain. specs/098-tools-preflight/tasks.md: tick T001, T024, T031, each confirmed against the tree (doc.go states both invariants T001 requires; toolslist_snapshot_test.go is labelled T024 and green; README carries all three FR-017 examples). T025/T027/T032/T033 are left unchecked — they are one-time process acts or did not survive verification. Deliberately NOT changed: no epics were added for shipped specs (084-100). roadmap.yaml is the working set by design and ROADMAP.md already renders a generated per-spec progress table, so absence of an epic is not drift. scripts/gen-roadmap.py --check-github: 3 warnings -> 0 errors, 0 warnings. * fix(roadmap): correct six claims flagged by cross-model review All six findings from the opencode gpt-5.6-sol review were independently re-verified against the tree before being applied; none were false positives. (The reviewer's initial claim that spec-gardener-auto-approve.yml is missing WAS a false positive from its glob tool and it self-corrected — the file exists at .github/workflows/spec-gardener-auto-approve.yml.) 1. Un-tick specs/098 T024. T024 asks for a golden snapshot test proving tools/list is BYTE-IDENTICAL across all three modes. Spec 099 (FR-014/015) later converted that test from a no-delta gate into an ENUMERATED-delta gate (toolsListAllowedDelta permits describe_tool and quarantine_security to move), so at HEAD the test no longer demonstrates the property T024 states. It may well have held on 098's own branch, but that is not checkable from the tree, and an unverifiable tick is exactly what this PR exists to remove. Badge 29/33 -> 28/33. 2. discovery-eval-harness note: drop the "SUPERSEDED — folded into token-bench-harness" framing. It contradicted itself once the epic was marked done — token-bench-harness is still unbuilt, so nothing could have been folded into it. Both halves shipped independently. 3. Same note: the eval.yml retrieval-d1 job is REPORT-ONLY on pull requests (continue-on-error, guarding against npx/uvx fetch flake) and blocks only on the nightly schedule. Promoting it to PR-blocking is still open (MCP-742). 4. Same note: the SC-003 recall@5 = 0.68 +/- 0.05 parity gate is asserted in bench/armindex_test.go, not armindex.go — the latter supplies the production index wiring, not the assertion. 5. token-bench-harness note: the literature-derived retry defaults live in bench/session.go (armRetryRates, research D8). bench/report.go only renders and labels them as estimates. The substantive point stands: retry rates are assumed, not observed. 6. token-bench-public note: "CI-wired" overstated the LiveMCPBench snapshot. bench.yml's regular job passes only -corpus-v2; the snapshot is runnable via -livemcptool but not run in CI, and its loader has no retrieval golden set, so it measures token/scale rather than recall. ToolRet does have its own bench.yml job. gen-roadmap.py --check and --check-github both still clean. * fix(roadmap): tighten two claims from cross-model review round 2 Both findings verified against the tree before applying. 1. The retrieval-D1 blocking scope was stated too narrowly. eval.yml sets continue-on-error only for github.event_name == 'pull_request', and the workflow triggers on pull_request, workflow_dispatch and schedule — so the job is report-only on PRs and BLOCKING on both nightly and manual dispatch runs, not "only on the nightly schedule". The note now says so, and records that the workflow's own inline comment carries the same omission. 2. discovery-eval-harness is now done but still declared depends_on: [token-bench-harness], which is unbuilt — a done node cannot depend on an open one, and the edge contradicted the corrected note saying both halves shipped independently. Edge removed; the id remains a valid depends_on target for anything that needs it. gen-roadmap.py --check and --check-github both clean. * fix(roadmap): scope four over-broad claims from cross-model review round 3 All four verified against GitHub and the tree before applying. 1. windows-tray: "no Windows-tray PR is open or merged" was false. #74 ("feature: windows tray support") merged 2025-10-23 and the cross-platform Go/systray build still ships on Windows. The claim now scopes to what the epic actually tracks — the native WebView2 replacement — for which no PR exists and native/windows/ is a README placeholder. 2. analytics-dashboard: "can never tick from code" conflated two things. A human can run spec 069's T023 Playwright sweep and tick the box; what can never exist is code evidence for it. Reworded so the distinction is explicit. 3. hygiene-spec-gardener: "three gardener runs ... auto-approved by #842" was wrong. Only #999 carries a github-actions APPROVED review. #824 has none and merged ~24 minutes BEFORE the auto-approve workflow landed; #870 records no approving review either. 4. token-bench-public: calling the MCP-specific third "integrated" overstated completion. ToolRet and LiveMCPTool are static retrieval/token corpora and lap-score is a linter — none is a public agent-loop task suite, which is what the task ultimately owes. Also noted for the record: the linux-arm64 Build Binaries failure on the previous commit is a flaky test, not a regression from this docs-only change. TestServiceStartScanDeepOnRunsSourceResolutionAndPass2 (internal/security/scanner) passes locally with -count=3, and two runs of this same branch four minutes apart — 33360429077 and 33360648947, both docs-only — disagreed on it. * fix(roadmap): ToolRet is not MCP-specific (cross-model review round 4) Verified: bench/corpusio/toolret.go documents ToolRet as a general tool-retrieval corpus whose documentation fields are "heterogeneous JSON/text, NOT MCP input schemas" (Schema stays nil). Grouping it under "the MCP-specific third" was wrong. The token-bench-public note now says public corpus/linter groundwork is integrated while NO public agent-loop task suite is, and names the ToolRet caveat explicitly so the next implementer does not re-inherit the error. Round 4 found no other issues; the round-3 windows-tray, analytics T023 and gardener corrections were all confirmed accurate against GitHub and the tree. * fix(roadmap): discovery-eval-harness is in_progress, not done (review round 5) The final review round caught an error in this PR's own flip, and it is the same failure mode the PR was written to fix: an epic marked done while a governing FR is unmet. Spec 065 FR-009 and SC-005 require CI to FAIL on a discovery regression beyond tolerance. The retrieval-D1 job sets continue-on-error for pull_request (D1 depends on npx/uvx fetches, a known flake source), so on the PR path CI does not fail — and eval.yml's own comment records the promotion to PR-blocking as still planned (MCP-742). The HARNESS shipped; the GATE did not. Reverted to in_progress and added the remaining work as a child task (discovery-eval-pr-blocking, MCP-742) so the epic's status has something in the DAG to point at. The reviewer also read CN-002 (frozen corpus, never live) as violated because D1 boots a live mcpproxy serving seven reference servers. Recorded as a weaker tension to adjudicate rather than a violation: #931 pinned all seven upstreams to freeze-era versions and the job gates on the exact corpus ID set, so the corpus is reproducible in practice. Whether that satisfies CN-002 or a committed snapshot is required is a judgement for the spec owner. gen-roadmap.py --check and --check-github both clean (0 errors, 0 warnings).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The break
The Retrieval regression gate (D1) fails on every PR that triggers it. The cause is outside this repo: the
mcpPython SDK cut 2.0.0 on 2026-07-28, which renamedMcpError→MCPErrorand removed the low-levelServer.list_tools/list_resourcesdecorators.Every uvx server in
specs/065-evaluation-foundation/datasets/snapshot-servers.config.jsondeclares only a lower bound (fetch mcp>=1.1.3,git mcp>=1.0.0,time mcp>=1.23.0,sqlite mcp[cli]>=1.6.0), so all four resolve 2.0.0 and all four are dead:fetch,time→ImportError: cannot import name 'McpError' … Did you mean: 'MCPError'?git,sqlite→AttributeError: 'Server' object has no attribute 'list_tools'The failing log's 24 tools confirms the mechanism exactly: the dataset has seven servers, not four. The three npx servers are healthy and expose 14 + 9 + 1 = 24; the four uvx servers expose 21 and are all down. 24 of 45 means the readiness loop (expects ≥ 44) times out and the job exits 1.
The fix
Every server pinned exactly, and the uvx ones additionally carry
--with mcp==1.29.0.Pinning the server packages alone would not have worked —
mcp-server-fetch==2026.7.10still resolvesmcp>=1.1.3→ 2.0.0. The break is transitive, so the constraint has to name the SDK. Both pins verified together, live.Exact
mcp==1.29.0rather thanmcp<2, deliberately:mcp<2fixes today and still floats within 1.x, which is the same latent flake one release later. 1.29.0 is the newest 1.x and is whatmcp<2resolves to today, so nothing is downgraded.All seven pinned, not just the broken four.
npx -y <pkg>with no@versionresolves latest on every run against a frozen corpus — identical exposure, just not yet triggered. Two of the non-fetch uvx servers were already failing with a different-looking error, so a fetch-only pin would have left the gate red.An unpinned dependency in a regression gate is a flake by construction: the baseline moves whenever an upstream publishes.
Verification
Reproduced the CI step verbatim (
retrieval-d1,.github/workflows/eval.yml:114-219) against the frozen corpus, golden set and baseline:Run twice, same result. The pinned versions reproduce the frozen 45-tool
corpus_v1tool for tool — names diffed, not just counts: filesystem 14, git 12, memory 9, sqlite 6, time 2, fetch 1, sequential-thinking 1.Caveat on what that proves: verified on macOS/arm64 with CPython 3.14; CI is ubuntu-latest at Python 3.11.13. The packages are pure-Python and the schemas platform-independent, so parity is expected — but the first CI run is the real check.
Residual not claimed as closed: transitive deps (httpx, pydantic, click) still float. A truly hermetic gate needs a lockfile, which a uvx-from-config invocation cannot express.
Two traps documented in the datasets README
mcpproxy serve --config <file>rewrites the file in place — alphabetised keys, all defaults materialised. A first local run turned a 7-line change into 140 insertions / 151 deletions on a tracked dataset file. The README previously told you to pointserve --configstraight at it..github/workflows/bench.yml:104) andbench/docker-compose.yml, so this one file fixes both gates.Also added an Upstream version pinning section recording the incident, the pin table, and the rule that bumping a pin is a corpus change — re-snapshot, diff, cut
corpus_v2; never editcorpus_v1(CN-002/FR-012).Scope
Two files: the dataset config and the datasets README. No Go code, no
go.modchange.