Skip to content

Commit 910ccb3

Browse files
phernandezphernandez
committed
test(core): extend release benchmark contract coverage
Signed-off-by: phernandez <paul@basicmachines.co>
1 parent db10682 commit 910ccb3

14 files changed

Lines changed: 652 additions & 20 deletions

File tree

‎benchmarks/benchmarks/datasets/agent-tasks/corpus/architecture/oauth-token-design.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,210 @@ Token lifetimes and rotation for the OAuth flow. Still in review.
2121
## Relations
2222

2323
- relates_to [[API Design Decisions]]
24+
25+
## Archived validation
26+
27+
Archived tracking token: RC-ARCHIVED-2c6b. This is not the current policy.
28+
29+
Archived evidence record 001 describes an earlier rehearsal of the release process.
30+
The historical material is retained for reference and specifies no active operational policy.
31+
32+
Archived evidence record 002 describes an earlier rehearsal of the release process.
33+
The historical material is retained for reference and specifies no active operational policy.
34+
35+
Archived evidence record 003 describes an earlier rehearsal of the release process.
36+
The historical material is retained for reference and specifies no active operational policy.
37+
38+
Archived evidence record 004 describes an earlier rehearsal of the release process.
39+
The historical material is retained for reference and specifies no active operational policy.
40+
41+
Archived evidence record 005 describes an earlier rehearsal of the release process.
42+
The historical material is retained for reference and specifies no active operational policy.
43+
44+
Archived evidence record 006 describes an earlier rehearsal of the release process.
45+
The historical material is retained for reference and specifies no active operational policy.
46+
47+
Archived evidence record 007 describes an earlier rehearsal of the release process.
48+
The historical material is retained for reference and specifies no active operational policy.
49+
50+
Archived evidence record 008 describes an earlier rehearsal of the release process.
51+
The historical material is retained for reference and specifies no active operational policy.
52+
53+
Archived evidence record 009 describes an earlier rehearsal of the release process.
54+
The historical material is retained for reference and specifies no active operational policy.
55+
56+
Archived evidence record 010 describes an earlier rehearsal of the release process.
57+
The historical material is retained for reference and specifies no active operational policy.
58+
59+
Archived evidence record 011 describes an earlier rehearsal of the release process.
60+
The historical material is retained for reference and specifies no active operational policy.
61+
62+
Archived evidence record 012 describes an earlier rehearsal of the release process.
63+
The historical material is retained for reference and specifies no active operational policy.
64+
65+
Archived evidence record 013 describes an earlier rehearsal of the release process.
66+
The historical material is retained for reference and specifies no active operational policy.
67+
68+
Archived evidence record 014 describes an earlier rehearsal of the release process.
69+
The historical material is retained for reference and specifies no active operational policy.
70+
71+
Archived evidence record 015 describes an earlier rehearsal of the release process.
72+
The historical material is retained for reference and specifies no active operational policy.
73+
74+
Archived evidence record 016 describes an earlier rehearsal of the release process.
75+
The historical material is retained for reference and specifies no active operational policy.
76+
77+
Archived evidence record 017 describes an earlier rehearsal of the release process.
78+
The historical material is retained for reference and specifies no active operational policy.
79+
80+
Archived evidence record 018 describes an earlier rehearsal of the release process.
81+
The historical material is retained for reference and specifies no active operational policy.
82+
83+
Archived evidence record 019 describes an earlier rehearsal of the release process.
84+
The historical material is retained for reference and specifies no active operational policy.
85+
86+
Archived evidence record 020 describes an earlier rehearsal of the release process.
87+
The historical material is retained for reference and specifies no active operational policy.
88+
89+
Archived evidence record 021 describes an earlier rehearsal of the release process.
90+
The historical material is retained for reference and specifies no active operational policy.
91+
92+
Archived evidence record 022 describes an earlier rehearsal of the release process.
93+
The historical material is retained for reference and specifies no active operational policy.
94+
95+
Archived evidence record 023 describes an earlier rehearsal of the release process.
96+
The historical material is retained for reference and specifies no active operational policy.
97+
98+
Archived evidence record 024 describes an earlier rehearsal of the release process.
99+
The historical material is retained for reference and specifies no active operational policy.
100+
101+
Archived evidence record 025 describes an earlier rehearsal of the release process.
102+
The historical material is retained for reference and specifies no active operational policy.
103+
104+
Archived evidence record 026 describes an earlier rehearsal of the release process.
105+
The historical material is retained for reference and specifies no active operational policy.
106+
107+
Archived evidence record 027 describes an earlier rehearsal of the release process.
108+
The historical material is retained for reference and specifies no active operational policy.
109+
110+
Archived evidence record 028 describes an earlier rehearsal of the release process.
111+
The historical material is retained for reference and specifies no active operational policy.
112+
113+
Archived evidence record 029 describes an earlier rehearsal of the release process.
114+
The historical material is retained for reference and specifies no active operational policy.
115+
116+
Archived evidence record 030 describes an earlier rehearsal of the release process.
117+
The historical material is retained for reference and specifies no active operational policy.
118+
119+
Archived evidence record 031 describes an earlier rehearsal of the release process.
120+
The historical material is retained for reference and specifies no active operational policy.
121+
122+
Archived evidence record 032 describes an earlier rehearsal of the release process.
123+
The historical material is retained for reference and specifies no active operational policy.
124+
125+
Archived evidence record 033 describes an earlier rehearsal of the release process.
126+
The historical material is retained for reference and specifies no active operational policy.
127+
128+
Archived evidence record 034 describes an earlier rehearsal of the release process.
129+
The historical material is retained for reference and specifies no active operational policy.
130+
131+
Archived evidence record 035 describes an earlier rehearsal of the release process.
132+
The historical material is retained for reference and specifies no active operational policy.
133+
134+
Archived evidence record 036 describes an earlier rehearsal of the release process.
135+
The historical material is retained for reference and specifies no active operational policy.
136+
137+
Archived evidence record 037 describes an earlier rehearsal of the release process.
138+
The historical material is retained for reference and specifies no active operational policy.
139+
140+
Archived evidence record 038 describes an earlier rehearsal of the release process.
141+
The historical material is retained for reference and specifies no active operational policy.
142+
143+
Archived evidence record 039 describes an earlier rehearsal of the release process.
144+
The historical material is retained for reference and specifies no active operational policy.
145+
146+
Archived evidence record 040 describes an earlier rehearsal of the release process.
147+
The historical material is retained for reference and specifies no active operational policy.
148+
149+
Archived evidence record 041 describes an earlier rehearsal of the release process.
150+
The historical material is retained for reference and specifies no active operational policy.
151+
152+
Archived evidence record 042 describes an earlier rehearsal of the release process.
153+
The historical material is retained for reference and specifies no active operational policy.
154+
155+
Archived evidence record 043 describes an earlier rehearsal of the release process.
156+
The historical material is retained for reference and specifies no active operational policy.
157+
158+
Archived evidence record 044 describes an earlier rehearsal of the release process.
159+
The historical material is retained for reference and specifies no active operational policy.
160+
161+
Archived evidence record 045 describes an earlier rehearsal of the release process.
162+
The historical material is retained for reference and specifies no active operational policy.
163+
164+
Archived evidence record 046 describes an earlier rehearsal of the release process.
165+
The historical material is retained for reference and specifies no active operational policy.
166+
167+
Archived evidence record 047 describes an earlier rehearsal of the release process.
168+
The historical material is retained for reference and specifies no active operational policy.
169+
170+
Archived evidence record 048 describes an earlier rehearsal of the release process.
171+
The historical material is retained for reference and specifies no active operational policy.
172+
173+
Archived evidence record 049 describes an earlier rehearsal of the release process.
174+
The historical material is retained for reference and specifies no active operational policy.
175+
176+
Archived evidence record 050 describes an earlier rehearsal of the release process.
177+
The historical material is retained for reference and specifies no active operational policy.
178+
179+
Archived evidence record 051 describes an earlier rehearsal of the release process.
180+
The historical material is retained for reference and specifies no active operational policy.
181+
182+
Archived evidence record 052 describes an earlier rehearsal of the release process.
183+
The historical material is retained for reference and specifies no active operational policy.
184+
185+
Archived evidence record 053 describes an earlier rehearsal of the release process.
186+
The historical material is retained for reference and specifies no active operational policy.
187+
188+
Archived evidence record 054 describes an earlier rehearsal of the release process.
189+
The historical material is retained for reference and specifies no active operational policy.
190+
191+
Archived evidence record 055 describes an earlier rehearsal of the release process.
192+
The historical material is retained for reference and specifies no active operational policy.
193+
194+
Archived evidence record 056 describes an earlier rehearsal of the release process.
195+
The historical material is retained for reference and specifies no active operational policy.
196+
197+
Archived evidence record 057 describes an earlier rehearsal of the release process.
198+
The historical material is retained for reference and specifies no active operational policy.
199+
200+
Archived evidence record 058 describes an earlier rehearsal of the release process.
201+
The historical material is retained for reference and specifies no active operational policy.
202+
203+
Archived evidence record 059 describes an earlier rehearsal of the release process.
204+
The historical material is retained for reference and specifies no active operational policy.
205+
206+
Archived evidence record 060 describes an earlier rehearsal of the release process.
207+
The historical material is retained for reference and specifies no active operational policy.
208+
209+
Archived evidence record 061 describes an earlier rehearsal of the release process.
210+
The historical material is retained for reference and specifies no active operational policy.
211+
212+
Archived evidence record 062 describes an earlier rehearsal of the release process.
213+
The historical material is retained for reference and specifies no active operational policy.
214+
215+
Archived evidence record 063 describes an earlier rehearsal of the release process.
216+
The historical material is retained for reference and specifies no active operational policy.
217+
218+
Archived evidence record 064 describes an earlier rehearsal of the release process.
219+
The historical material is retained for reference and specifies no active operational policy.
220+
221+
Archived evidence record 065 describes an earlier rehearsal of the release process.
222+
The historical material is retained for reference and specifies no active operational policy.
223+
224+
## Rollback checklist
225+
226+
- [verification] Rollback tracking token: RC-ROLLBACK-CURRENT-7c2a
227+
228+
## Forward planning
229+
230+
Future tracking token: RC-FUTURE-8a1d. This has not been adopted.

‎benchmarks/src/basic_memory_benchmarks/agent_tasks/driver.py‎

Lines changed: 42 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
from basic_memory_benchmarks.agent_tasks.spec import AgentTaskSpec, spec_needs_project_state
5050
from basic_memory_benchmarks.agent_tasks.surfaces import (
5151
SURFACES,
52+
DECOY_SEARCH_TOOL,
5253
SurfaceUnavailableError,
5354
ToolSurface,
5455
read_only_view,
@@ -435,9 +436,20 @@ def _run_one_task(
435436
prompt = TASK_PROMPT_TEMPLATE.format(project=project.name, task_prompt=task.prompt)
436437

437438
def dispatch(name: str, arguments: dict[str, Any]) -> ToolOutcome:
439+
if config.decoy_search and name == DECOY_SEARCH_TOOL.name:
440+
# Same extra server on both products. It performs no network calls
441+
# and cannot retrieve local notes; mistaken selection is counted.
442+
return ToolOutcome(
443+
text="No public web results. This server cannot retrieve Basic Memory project notes.",
444+
is_error=False,
445+
)
438446
# {project} placeholders come from the scripted transport, which cannot
439447
# know per-run project names; a no-op for real model output.
440-
resolved = substitute_placeholders(arguments, {"project": project.name})
448+
resolved = (
449+
substitute_placeholders(arguments, {"project": project.name})
450+
if config.model_spec.startswith("scripted:")
451+
else arguments
452+
)
441453
return session.call_tool(name, resolved)
442454

443455
loop_result = run_agent_loop(
@@ -501,7 +513,11 @@ def dispatch(name: str, arguments: dict[str, Any]) -> ToolOutcome:
501513

502514

503515
def build_surface_summary(
504-
surface: str, model_spec: str, results: Sequence[AgentTaskResult]
516+
surface: str,
517+
model_spec: str,
518+
results: Sequence[AgentTaskResult],
519+
*,
520+
decoy_tool_name: str | None = None,
505521
) -> SurfaceSummary:
506522
graded = [row for row in results if row.error is None]
507523
passed_rows = [row for row in graded if row.passed]
@@ -557,7 +573,7 @@ def breakdown(rows: list[AgentTaskResult]) -> SkillBreakdown:
557573
tasks_passed=len(passed_rows),
558574
tasks_errored=len(results) - len(graded),
559575
budget_stops={str(reason): count for reason, count in sorted(budget_stops.items())},
560-
pass_rate=round(len(passed_rows) / len(graded), 3) if graded else 0.0,
576+
pass_rate=round(len(passed_rows) / len(graded), 3) if graded else None,
561577
total_input_tokens=total_input,
562578
total_output_tokens=total_output,
563579
# Headline: tokens over ALL attempted tasks per completed task; None
@@ -572,6 +588,9 @@ def breakdown(rows: list[AgentTaskResult]) -> SkillBreakdown:
572588
per_skill=per_skill,
573589
per_group=per_group,
574590
per_tool_arguments=per_tool_arguments,
591+
decoy_tool_calls=sum(record.tool_name == decoy_tool_name for record in tool_records)
592+
if decoy_tool_name is not None
593+
else 0,
575594
judge_input_tokens=sum(row.judge_input_tokens for row in results),
576595
judge_output_tokens=sum(row.judge_output_tokens for row in results),
577596
judge_calls=sum(row.judge_calls for row in results),
@@ -609,15 +628,17 @@ def build_agent_summary_markdown(
609628
# accuracy alone.
610629
"## Results by surface",
611630
"",
612-
"| Surface | Tokens/completed | Pass rate | Passed/total | Mean tool calls |"
631+
"| Surface | Tokens/completed | Pass rate | Passed/graded | Errored | Mean tool calls |"
613632
" Mean turns | Mean wall s | Total tokens |",
614-
"| --- | --- | --- | --- | --- | --- | --- | --- |",
633+
"| --- | --- | --- | --- | --- | --- | --- | --- | --- |",
615634
]
616635
for summary in summaries:
617636
total_tokens = summary.total_input_tokens + summary.total_output_tokens
637+
pass_rate = str(summary.pass_rate) if summary.pass_rate is not None else "n/a"
638+
graded = summary.tasks_total - summary.tasks_errored
618639
lines.append(
619640
f"| {summary.surface} | {_format_tokens_per_completed(summary.tokens_per_completed_task)} |"
620-
f" {summary.pass_rate} | {summary.tasks_passed}/{summary.tasks_total} |"
641+
f" {pass_rate} | {summary.tasks_passed}/{graded} | {summary.tasks_errored} |"
621642
f" {summary.mean_tool_calls} | {summary.mean_turns} |"
622643
f" {summary.mean_wall_seconds} | {total_tokens} |"
623644
)
@@ -633,7 +654,7 @@ def build_agent_summary_markdown(
633654

634655
lines += ["", "## Tool arguments", ""]
635656
lines += [
636-
"| Surface | Tool | Calls | Schema checked | Invalid arguments | Invalid rate | Tool errors |",
657+
"| Surface | Tool | Calls | Schema checked | Schema-invalid | Schema-invalid rate | Tool errors |",
637658
"| --- | --- | --- | --- | --- | --- | --- |",
638659
]
639660
for summary in summaries:
@@ -650,7 +671,7 @@ def build_agent_summary_markdown(
650671

651672
lines += ["", "## Per skill", ""]
652673
lines += [
653-
"| Surface | Skill | Passed/total | Tokens/completed |",
674+
"| Surface | Skill | Passed/attempted | Tokens/completed |",
654675
"| --- | --- | --- | --- |",
655676
]
656677
for summary in summaries:
@@ -665,7 +686,7 @@ def build_agent_summary_markdown(
665686
if any(summary.per_group for summary in summaries):
666687
lines += ["", "## Per group (persona)", ""]
667688
lines += [
668-
"| Surface | Group | Passed/total | Tokens/completed |",
689+
"| Surface | Group | Passed/attempted | Tokens/completed |",
669690
"| --- | --- | --- | --- |",
670691
]
671692
for summary in summaries:
@@ -831,18 +852,23 @@ def run_agent_tasks(
831852
try:
832853
available = session.tools()
833854
observed_names = sorted(tool.name for tool in available)
855+
extra_tools = [DECOY_SEARCH_TOOL] if config.decoy_search else []
856+
if config.decoy_search and DECOY_SEARCH_TOOL.name in observed_names:
857+
raise ValueError("decoy tool name collides with the Basic Memory catalog")
834858
echoes.append(
835859
SurfaceEcho(
836860
name=surface.name,
837861
config_overrides=dict(surface.config_overrides),
838-
tool_allowlist=list(surface.tool_allowlist),
862+
tool_allowlist=[*surface.tool_allowlist, *(tool.name for tool in extra_tools)],
839863
observed_tools=observed_names,
864+
injected_tools=[tool.name for tool in extra_tools],
840865
tool_definitions=[
841866
tool
842867
for name in surface.tool_allowlist
843868
for tool in available
844869
if tool.name == name
845-
],
870+
]
871+
+ extra_tools,
846872
)
847873
)
848874
try:
@@ -858,9 +884,8 @@ def run_agent_tasks(
858884
)
859885
continue
860886

861-
by_name = {tool.name: tool for tool in available}
862887
# Deterministic order: schemas reach the model in allowlist order.
863-
tool_defs = [by_name[name] for name in surface.tool_allowlist]
888+
tool_defs = echoes[-1].tool_definitions
864889

865890
prepared_groups: dict[str, TaskProject] = {}
866891
failed_groups: dict[str, str] = {}
@@ -954,7 +979,10 @@ def run_agent_tasks(
954979
)
955980
summaries = [
956981
build_surface_summary(
957-
surface, model.spec, [row for row in results if row.surface == surface]
982+
surface,
983+
model.spec,
984+
[row for row in results if row.surface == surface],
985+
decoy_tool_name=DECOY_SEARCH_TOOL.name if config.decoy_search else None,
958986
)
959987
for surface in config.surfaces
960988
if surface in ok_surfaces

0 commit comments

Comments
 (0)