Repository navigation
The testing → test-servers hand-off fires only a third of the time #2247
Description
Now that #2204 can measure a hand-off, the first thing it measures is a weak one.
testing's body opens by telling the model that picking a fixture is /test-servers and that it has to load it. With the two committed chain cases in .claude/skills/test-servers/evals/evals.json, that pointer is followed 33% of the time (RUNS=3, CHAIN_MAX_TURNS=14):
Hand-off (14 turns)
FAIL 33% testing → test-servers Write an integration test that exercises tool listing end to end.
FAIL 33% testing → test-servers Add end-to-end coverage for the tool-list pagination path.
Two things worth knowing before working on this:
- An earlier draft of those prompts measured 100% and 67%, and that was an artefact. They said "against a real server" / "against a live server", which is
test-servers' own trigger — so the model could picktestingand then picktest-serversfrom the original prompt, in that order, scoring a hit that would have survived deleting the pointer entirely. The prompts were rewritten to carry no server cue, and 33% is what the pointer alone is worth. Any fix has to be re-measured against prompts with no target cue, or it measures the same artefact. - Do not fix this by lowering
CHAIN_THRESHOLD. The bar is 0.5 ("the pointer is taken more often than not") and the two red cases are the standing measurement, not a build break —skills:evalis deliberately not a gate.
What to try
- Reshape where and how
testingnames/test-servers. Tune skill descriptions and eval cases to a reliable trigger rate, and document how #2202 showed that the shape of a description is the lever that moves a trigger rate; the same may hold for a pointer inside a body. - Check whether the run reaches a point where it knows it needs a fixture at all within 14 turns —
CHAIN_MAX_TURNSis a knob, and a rate that climbs with it means the budget is the constraint rather than the pointer. - Re-run the full suite afterwards, not just these two cases: per AGENTS.md, changing what one skill says measurably moves the trigger rates of skills nobody touched.
Acceptance
- The
testing → test-servershand-off measures above 50% atRUNS=5, with prompts that carry notest-serverstrigger of their own. - The full
skills:evalsuite shows no first-move regression.
Activity
feat(skills): address Copilot review round 1 on #2204
cliffhall commentedon Sep 6, 2026
Shipped in #2261, merged to
v2/mainas0d947ad5.Acceptance, against this issue's own criteria:
RUNS=5, prompts carrying notest-serverstriggerCHAIN_THRESHOLDnot loweredThe two prompts were not touched, so this is not the artefact the issue warns about.
CHAIN_MAX_TURNSalso stayed at 14 — which answers the second suggestion: the turn budget was never the constraint, the pointer was.What moved the number: reshaping the pointer from a classification ("anything that needs a real server is
/test-servers") into an imperative step with a trigger ("load thetest-serversskill now — that is step one"), and repeating it at the two later points where a run actually establishes it needs a fixture. The top of a body is read before the model knows it needs the second skill, so a pointer living only there has been scrolled past by the time it matters.docs/skill-authoring.mdsays so explicitly rather than claiming either as independently causal.Neither description was touched, so the listing budget is unchanged at 3,234/4,000.
Copilot ran eight review rounds on the PR. Most of them were about the trigger condition rather than the pointer, and it moved tier → folder → connecting → build-ran → uses a
test-servers/fixture as each proxy was shown to diverge from the real condition in a documented case in this repo. A ninth round landed after the merge; see the follow-up issue.chore: land the #2247 round-9 review findings on v2/main (#2322)