docs(core): record funded Haiku pilot and mutation verification

Signed-off-by: phernandez <paul@basicmachines.co>

Author: phernandez

benchmarks/benchmarks/results/1.0-rc-pilot-2026-10-28.md

@@ -0,0 +1,70 @@
+# 1.0 RC tool comparison: first Haiku pilot
+
+The three shared task goals passed on both versions. Across this small pilot,
+the 1.0 catalog used **57.5% fewer agent tokens** and **three tool calls versus
+five**. Treat this as pilot evidence; repeated runs and broader task/model
+coverage remain necessary for a release recommendation.
+
+| Task goal | v0.24.1 tokens | 1.0 RC tokens | v0.24.1 calls | 1.0 RC calls |
+| --- | ---: | ---: | ---: | ---: |
+| List one folder | 44,294 | 22,575 | 1 | 1 |
+| Read by title | 42,534 | 20,801 | 1 | 1 |
+| Write at an exact path | 65,014 | 21,108 | 3 | 1 |
+| Total | 151,842 | 64,484 | 5 | 3 |
+
+Tokens per completed task: **50,614 → 21,494.7**. Counts include all reported
+agent input/output tokens. All tool calls conformed to their advertised schemas;
+there were no tool errors or model/harness errors in these funded runs.
+
+The legacy write used `write_note`, `move_note`, then `read_note`. The 1.0 write
+used `bm_write`. Both produced the required file, frontmatter and observation,
+preserving existing notes. The legacy write stopped on the token budget before
+a final assistant reply; its file-state goal still passed. The 60,000-token
+budget is checked before each model call, so a call can overshoot it (65,014
+tokens here). The other five tasks ended with final replies. The new read/list
+graders require a successful matching tool call as well as the correct answer.
+
+## Reproducibility
+
+- Baseline: v0.24.1, `1d0e09b6f4876c3f339c30e8dbfcf66e59e2c356`.
+- Candidate: main/1.0 RC, `0884e204d84ac9d18124ad78601a80925643f901`.
+- Harness: `9e4d2beb550b7741882946a18967e1b2db8bf072`.
+- Model requested and returned: `claude-haiku-5-5`, Anthropic's OpenAI-compatible endpoint.
+- Temperature omitted; eight turns and 60,000 pre-call tokens per task; 180-second task budget.
+- Full catalogs: 27 legacy tools versus 12 bm tools, alphabetical schema order.
+- Fresh isolated project/corpus per task, same prompts and deterministic graders.
+- Product-default semantic settings on both products; SQLite; no Redis configuration.
+- Corpus: 25 notes, SHA-256 `b91f5fe80849e2774b860b0b4a59d678eff2820201029570ed716fac9df12884`.
+- Task specs: SHA-256 `985923496748505e8f3246aaf320343031fc922cd7ade07f1e3e9da68510ed28`.
+
+Raw manifests include every tool schema/description supplied to the model.
+Per-turn artifacts retain returned model IDs, tokens, calls, result sizes and
+error information. Supplementary `product-runtime.json` files record target
+Python, installed package versions and lock hashes; both products use Python
+3.14.8. `pilot-input.py` retains the local runner without credentials.
+
+Run artifacts, relative to the Core repository:
+
+- `benchmarks/benchmarks/runs/release-haiku55-pilot-legacy-02/`
+- `benchmarks/benchmarks/runs/release-haiku55-pilot-bm-02/`
+
+The earlier legacy `-01` attempt contains billing-rejected model-error rows and
+is excluded from this comparison. Host contention affected this session, so
+timing conclusions require the separate controlled performance workload.
+
+## Scripted mutation verification
+
+All **20** real-MCP mutation checks passed: ten goals on each product. They cover
+append, prepend, exact replacement, section replacement, before/after-section
+insertion, metadata-only change, move, delete and stale-checksum refusal.
+Each expected mutation was checked against the resulting files; unrelated notes
+remained byte-identical. Body comparisons preserve exact nonblank lines in
+order and ignore blank-line layout. Both stale-checksum calls produced an
+actual conflict refusal and left all note bytes unchanged.
+
+These are plumbing/correctness checks using scripted inputs, with embeddings
+explicitly disabled. Their canned answers and fake token counts are excluded
+from the real-model token comparison above. Artifacts:
+
+- `benchmarks/benchmarks/runs/release-mutation-legacy-01/`
+- `benchmarks/benchmarks/runs/release-mutation-bm-01/`

benchmarks/docs/1.0-release-comparison-plan.md

@@ -55,7 +55,7 @@ otherwise idle host. Separate cold startup, warm operation and indexing time.
 - [x] Add read by title/path/permalink/memory URL.
 - [ ] Add bounded long-note reads.
 - [x] Define exact-path write, all seven edit goals, move/delete and revision-conflict tasks.
-- [ ] Prove the new mutation goals on both products through real MCP.
+- [x] Prove the new mutation goals on both products through real MCP.
 - [ ] Add search scope/filters, folder listing and recursive finding.
 - [ ] Add project selection/ambiguity, recent activity and project identifiers.
 - [ ] Add refusal recovery and a decoy search tool from another server.
@@ -197,6 +197,18 @@ expanding expensive datasets or backend matrices.
   Model-returned identifiers and a deterministic task-spec checksum are now
   recorded. Scripted mutation validation will explicitly disable embeddings;
   scored agent runs retain product defaults unless a profile is stated.
+- User topped up Anthropic; a generation preflight succeeded. Fresh funded
+  Haiku pilot runs (`release-haiku55-pilot-{legacy,bm}-02`) passed all three
+  task goals on both full catalogs. Mean agent tokens per completed task:
+  50,614 legacy versus 21,494.7 bm (57.5% lower in this small pilot). Tool calls:
+  five versus three. Legacy exact-path write reached the correct file state
+  and then stopped at its pre-call token budget, without a final reply. No
+  schema-invalid calls or tool errors. Full result and limitations:
+  `benchmarks/benchmarks/results/1.0-rc-pilot-2026-10-28.md`.
+- Scripted mutation verification completed: ten goals on each product, 20/20
+  passed, all arguments schema-valid and only the deliberately stale-checksum
+  calls refused. Model/harness source remained pinned to `9e4d2beb5` during
+  these runs. Billing is resolved; host contention still limits timing claims.
 
 ## Reproduce the scripted smoke