Skip to content

fix(mcp): say what recall's max_tokens counts, and return compact JSON - #4870

Merged
nicoloboschi merged 3 commits into
vectorize-io:mainfrom
JayOfTheKeyboard:fix/mcp-recall-max-tokens-scope
Oct 6, 2026
Merged

nicoloboschi merged 3 commits into
vectorize-io:mainfrom
JayOfTheKeyboard:fix/mcp-recall-max-tokens-scope

Conversation

@JayOfTheKeyboard

Copy link
Copy Markdown
Contributor

The MCP recall tool describes max_tokens as the "Maximum tokens to return in results". The engine counts only each fact's text, as MemoryEngine.recall_async's docstring says ("counts only 'text' field"). Every result also carries its id, tags, scores, dates, document and chunk ids. An agent that sizes its context from the tool description is off by a wide margin.

Measured on 0.10.1 against a real bank (~900 facts), with budget: "low" and the default max_tokens: 4096, two queries returned:

query 1 query 2
results 117 90
fact text 19,270 chars (~4,800 tokens) 17,582 chars (~4,400 tokens)
whole response 119,494 chars 87,975 chars
of which indent=2 whitespace 26,693 20,270

So the budget works as the engine intends. The response is still about six times what the tool promises, and Claude Code refused both results as over its tool-output limit.

Fix

  • Docstring, both copies: max_tokens is now described as the token budget for the facts' text. It says the per-result metadata is extra, so the response is several times larger.
  • The bank-id copy returns compact JSON (model_dump_json() instead of model_dump_json(indent=2)). The session-bank copy returns a dict, which FastMCP already serialises compact (pydantic_core.to_json with no indent), so the two copies now send the same shape. This removes about a fifth of the response. Callers already json.loads this string.

Not changed: what max_tokens counts, and which fields a result carries. Either would change the engine's contract or the payload, and that is your call. For example, a separate budget covering the whole response, or leaving out scores unless asked for, could close the rest of the gap. I'm happy to follow up with either.

Test

TestRecallResponseSize in tests/test_mcp_tools.py:

  • test_bank_id_recall_returns_compact_json uses a real RecallResult. It checks that the output parses to the same payload and contains no newline.
  • test_recall_docs_say_max_tokens_counts_fact_text_only is parametrized over both copies.

All three fail on the parent commit. With only the docstring fixed, the compact test still fails.

tests/test_mcp_*.py pass (436 on current main). The full api-slim suite (without the hs_llm_* markers): 10,020 passed. The 5 failures and 1 error are in test_fact_extraction_output_ratio.py / test_fact_extraction_analysis.py, which need HINDSIGHT_API_LLM_API_KEY and fail the same way on main. Ruff and ty are unchanged from main. The two pre-existing F841s in test_mcp_tools.py and the 15 ty diagnostics are on main too.

@github-actions github-actions Bot added the core label Sep 28, 2026
JayOfTheKeyboard and others added 3 commits October 6, 2026 11:06
The MCP recall tool described max_tokens as the "maximum tokens to
return in results". The engine counts only each fact's text, as its own
docstring says, while every result also carries its id, tags, scores and
dates. At the default 4096, a recall against a real bank returned about
4,500 tokens of fact text inside an 88-119k character response, so an
agent sizing its context from the tool description was off by about
six times.

The bank-id copy of the tool also pretty-printed its JSON (indent=2),
about a fifth of that response, while FastMCP already sends the
session-bank copy's dict compact. It is now compact too.
…elper

- mcp-server.md still said max_tokens was the maximum tokens returned
- TestRecallResponseSize._result gets a return type; import moved to module top
@nicoloboschi
nicoloboschi force-pushed the fix/mcp-recall-max-tokens-scope branch from 4a9ad03 to 67a76a0 Compare October 6, 2026 09:08
@nicoloboschi
nicoloboschi merged commit 1a61579 into vectorize-io:main Oct 6, 2026
333 of 337 checks passed
@JayOfTheKeyboard

Copy link
Copy Markdown
Contributor Author

Thanks for fixing the docs line I missed, and for the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants