Skip to content

Commit 8c68372

Browse files
roaminroMastra Code (anthropic/claude-opus-4-8)
andauthored
feat(core): add autoLoad and opt-in context storage to ToolSearchProcessor (#17691)
Co-authored-by: Mastra Code (anthropic/claude-opus-4-8) <noreply@mastra.ai>
1 parent 30d9d71 commit 8c68372

6 files changed

Lines changed: 973 additions & 274 deletions

File tree

.changeset/yellow-oranges-shine.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
'@mastra/core': minor
3+
---
4+
5+
Added two options to `ToolSearchProcessor` for cheaper, more cache-friendly tool discovery.
6+
7+
**`autoLoad`: one-turn discovery**
8+
9+
Set `search.autoLoad` so that `search_tools` activates the matching tools immediately, removing the separate `load_tool` round-trip. The model searches once and can call a discovered tool on its next turn instead of searching, loading, then calling. This cuts one model turn (and a full prompt replay) per discovery.
10+
11+
```typescript
12+
import { ToolSearchProcessor } from '@mastra/core/processors';
13+
14+
const toolSearch = new ToolSearchProcessor({
15+
tools: allTools,
16+
search: { topK: 3, autoLoad: true },
17+
});
18+
```
19+
20+
**`storage`: opt-in `'context'` mode**
21+
22+
Choose where loaded-tool state lives. The default `'in-memory'` keeps the original behavior. The new opt-in `'context'` mode derives loaded tools from the conversation messages, so it is restart-safe, needs no memory configuration, and de-loads a tool once its discovery result is no longer in the messages.
23+
24+
```typescript
25+
const toolSearch = new ToolSearchProcessor({
26+
tools: allTools,
27+
storage: 'context',
28+
});
29+
```
30+
31+
Both modes are cache-friendly when loading tools, since loads are append-only and keep the cached prompt prefix stable. The default `'in-memory'` store still shares a single `'default'` entry across anonymous (no thread ID) requests; use `storage: 'context'` to keep anonymous requests isolated and derive loaded tools from the conversation messages.

docs/src/content/en/reference/processors/tool-search-processor.mdx

Lines changed: 119 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ const toolSearch = new ToolSearchProcessor({
5050
},
5151
{
5252
name: 'search',
53-
type: '{ topK?: number; minScore?: number }',
53+
type: '{ topK?: number; minScore?: number; autoLoad?: boolean }',
5454
description: 'Configuration for the search behavior.',
5555
isOptional: true,
5656
},
@@ -68,11 +68,27 @@ const toolSearch = new ToolSearchProcessor({
6868
isOptional: true,
6969
default: '0',
7070
},
71+
{
72+
name: 'search.autoLoad',
73+
type: 'boolean',
74+
description:
75+
'When true, tools returned by search_tools are activated immediately as part of the search. The load_tool meta-tool is not exposed, collapsing the two-step search then load flow into a single search step. Discovered tools become available on the next turn. Keep topK conservative since every match is activated.',
76+
isOptional: true,
77+
default: 'false',
78+
},
79+
{
80+
name: 'storage',
81+
type: "'in-memory' | 'context'",
82+
description:
83+
"Where loaded-tool state lives. 'in-memory' (default) tracks loaded tools in an in-memory map per thread with TTL cleanup (see ttl); state is lost on restart and anonymous requests share a 'default' entry. 'context' derives loaded state from the conversation messages — a tool is loaded while a search_tools/load_tool result naming it remains in the messages; it is restart-safe, requires no memory, and de-loads automatically once that result is no longer present in the messages. The 'context' store is opt-in.",
84+
isOptional: true,
85+
default: "'in-memory'",
86+
},
7187
{
7288
name: 'ttl',
7389
type: 'number',
7490
description:
75-
'Time-to-live for thread state in milliseconds. After this duration of inactivity, thread state will be cleaned up. Set to 0 to disable cleanup.',
91+
"Time-to-live for in-memory thread state, in milliseconds. Only applies to the default storage: 'in-memory' store; after this duration of inactivity thread state is cleaned up. Set to 0 to disable cleanup. Ignored by the 'context' store.",
7692
isOptional: true,
7793
default: '3600000',
7894
},
@@ -116,6 +132,48 @@ const toolSearch = new ToolSearchProcessor({
116132
]}
117133
/>
118134

135+
## Methods
136+
137+
### State inspection (legacy `'in-memory'` store)
138+
139+
These methods operate on the default `'in-memory'` store only. They are no-ops for the `'context'` store, whose state lives in the conversation messages rather than an in-process map.
140+
141+
#### `clearState(threadId)`
142+
143+
Clears the loaded-tool state for a single thread.
144+
145+
```typescript
146+
processor.clearState('thread-123')
147+
```
148+
149+
#### `clearAllState()`
150+
151+
Clears loaded-tool state for every thread.
152+
153+
```typescript
154+
processor.clearAllState()
155+
```
156+
157+
#### `getStateStats()`
158+
159+
Returns the number of tracked threads and the oldest access time, for debugging in-memory growth.
160+
161+
```typescript
162+
const { threadCount, oldestAccessTime } = processor.getStateStats()
163+
```
164+
165+
Returns: `{ threadCount: number; oldestAccessTime: number | null }`
166+
167+
#### `cleanupNow()`
168+
169+
Immediately runs TTL cleanup instead of waiting for the scheduled sweep.
170+
171+
```typescript
172+
const cleaned = processor.cleanupNow()
173+
```
174+
175+
Returns: `number` — the count of threads cleaned up.
176+
119177
## Request-aware filtering
120178

121179
Use `filter` to apply request-specific policy to dynamic tools. The hook receives the resolved tool ID as `toolName`, the tool, request context, and phase. `toolName` is the ID returned by `search_tools`, which may differ from the key used in the `tools` object.
@@ -143,7 +201,7 @@ The `phase` value describes where the filter is being applied:
143201
- `load`: Blocks `load_tool` from loading disallowed tools.
144202
- `active`: Hides already-loaded tools from the current request if they are no longer allowed.
145203

146-
If the hook throws or rejects, `ToolSearchProcessor` treats the tool as disallowed for that request. The hook may run for every matching search candidate, so keep async policy checks cheap or cached. The meta-tools `search_tools` and `load_tool` are always available. Tools passed directly through the agent or `processInputStep` remain available unless you filter them outside `ToolSearchProcessor`.
204+
If the hook throws or rejects, `ToolSearchProcessor` treats the tool as disallowed for that request. The hook may run for every matching search candidate, so keep async policy checks cheap or cached. The `search_tools` meta-tool is always available; `load_tool` is available unless `search.autoLoad` is enabled. Tools passed directly through the agent or `processInputStep` remain available unless you filter them outside `ToolSearchProcessor`.
147205

148206
## Extended usage example
149207

@@ -185,6 +243,64 @@ The agent workflow is:
185243
1. The loaded tool becomes available on the next turn
186244
1. Agent uses the loaded tool normally
187245

246+
## Single-step discovery with `autoLoad`
247+
248+
Set `search.autoLoad` to `true` to skip the separate load step. The tools returned by `search_tools` are activated immediately, and the `load_tool` meta-tool is not exposed. This removes one model turn per discovery, which lowers token usage and latency, and works the same across providers.
249+
250+
```typescript
251+
const toolSearch = new ToolSearchProcessor({
252+
tools: allTools,
253+
search: {
254+
topK: 3,
255+
autoLoad: true,
256+
},
257+
})
258+
```
259+
260+
With `autoLoad` the workflow becomes:
261+
262+
1. Agent receives a user message
263+
1. Agent calls `search_tools` with keywords
264+
1. The matching tools are activated automatically and become available on the next turn
265+
1. Agent uses the tool normally
266+
267+
Every match is activated, so keep `topK` small (for example, `3`) to avoid adding tools the agent did not need. Activated tools are appended after existing tools, which keeps the cached prompt prefix stable for providers that support prompt caching.
268+
269+
## Loaded-tool storage
270+
271+
The `storage` option controls where the set of loaded tools is tracked. The default is `'in-memory'`; the `'context'` store is opt-in.
272+
273+
### `'in-memory'` (default)
274+
275+
Loaded tools are tracked in an in-memory map per thread, with TTL-based cleanup controlled by the `ttl` option (default one hour). This is the original behavior:
276+
277+
- Requires no memory configuration.
278+
- State is lost on process restart.
279+
- Requests with no thread ID share a single `'default'` entry.
280+
281+
Use `clearState`, `clearAllState`, `getStateStats`, and `cleanupNow` to inspect or reset this store.
282+
283+
### `'context'`
284+
285+
Loaded state is derived from the conversation messages: a tool is loaded while a `search_tools` or `load_tool` result naming it remains in the messages. This mode:
286+
287+
- Requires no memory configuration.
288+
- Is restart-safe — the durable record is the persisted message history.
289+
- De-loads a tool automatically once that result is no longer present in the messages.
290+
291+
```typescript
292+
import { ToolSearchProcessor } from '@mastra/core/processors'
293+
294+
const toolSearch = new ToolSearchProcessor({
295+
tools: allTools,
296+
storage: 'context',
297+
})
298+
```
299+
300+
Loading tools is cache-friendly in both modes: loads are append-only, so the cached prompt prefix stays stable for providers that support prompt caching.
301+
302+
Unloading a tool changes the tool definitions sent to the model, which shifts the cached prefix and causes the next turn to pay a cache write instead of a cache hit. In `'in-memory'` mode this happens when a thread's state is evicted by `ttl`. In `'context'` mode it happens when a tool's discovery result is no longer present in the messages (for example, when older messages are trimmed) — the tool de-loads and the model must search for it again before reuse. This is expected: removing an unused tool trades one cache write for a smaller prefix on later turns.
303+
188304
## Combining with other processors
189305

190306
```typescript
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
import { describe, it, expect } from 'vitest';
2+
3+
import type { ProcessInputStepArgs } from '../index';
4+
import { deriveLoadedNamesFromMessages, LegacyMapLoadedToolStore, ContextLoadedToolStore } from './tool-search-stores';
5+
6+
/**
7+
* Build a minimal ProcessInputStepArgs carrying conversation messages with the
8+
* given search_tools / load_tool tool-invocation results.
9+
*/
10+
function argsWithMessages(
11+
invocations: Array<{ toolName: 'search_tools' | 'load_tool'; result: unknown }>,
12+
): ProcessInputStepArgs {
13+
return {
14+
messages: [
15+
{
16+
id: 'm1',
17+
role: 'assistant',
18+
content: {
19+
format: 2,
20+
parts: invocations.map((inv, i) => ({
21+
type: 'tool-invocation' as const,
22+
toolInvocation: {
23+
state: 'result' as const,
24+
toolCallId: `call-${i}`,
25+
toolName: inv.toolName,
26+
args: {},
27+
result: inv.result,
28+
},
29+
})),
30+
},
31+
},
32+
],
33+
} as unknown as ProcessInputStepArgs;
34+
}
35+
36+
describe('deriveLoadedNamesFromMessages', () => {
37+
it('reads names from a search_tools result (results[].name)', () => {
38+
const args = argsWithMessages([
39+
{ toolName: 'search_tools', result: { results: [{ name: 'weather' }, { name: 'calendar' }] } },
40+
]);
41+
expect([...deriveLoadedNamesFromMessages(args)].sort()).toEqual(['calendar', 'weather']);
42+
});
43+
44+
it('reads names from a load_tool result (loaded[])', () => {
45+
const args = argsWithMessages([{ toolName: 'load_tool', result: { loaded: ['github_create_issue'] } }]);
46+
expect([...deriveLoadedNamesFromMessages(args)]).toEqual(['github_create_issue']);
47+
});
48+
49+
it('unions across multiple invocations and ignores other tools', () => {
50+
const args = argsWithMessages([
51+
{ toolName: 'search_tools', result: { results: [{ name: 'weather' }] } },
52+
{ toolName: 'load_tool', result: { loaded: ['calendar'] } },
53+
]);
54+
expect([...deriveLoadedNamesFromMessages(args)].sort()).toEqual(['calendar', 'weather']);
55+
});
56+
57+
it('returns empty when messages are missing', () => {
58+
expect(deriveLoadedNamesFromMessages({} as ProcessInputStepArgs).size).toBe(0);
59+
});
60+
});
61+
62+
describe('LegacyMapLoadedToolStore', () => {
63+
const emptyArgs = argsWithMessages([]);
64+
65+
it('tracks loaded tools per thread', () => {
66+
const store = new LegacyMapLoadedToolStore({ ttl: 0 });
67+
store.addLoaded(['weather'], { threadId: 'thread-1', args: emptyArgs });
68+
store.addLoaded(['calendar'], { threadId: 'thread-2', args: emptyArgs });
69+
70+
expect([...store.getLoadedNames({ threadId: 'thread-1', args: emptyArgs })]).toEqual(['weather']);
71+
expect([...store.getLoadedNames({ threadId: 'thread-2', args: emptyArgs })]).toEqual(['calendar']);
72+
});
73+
74+
it('shares the default entry across anonymous requests (original behavior)', () => {
75+
const store = new LegacyMapLoadedToolStore({ ttl: 0 });
76+
store.addLoaded(['weather'], { threadId: undefined, args: emptyArgs });
77+
expect([...store.getLoadedNames({ threadId: undefined, args: emptyArgs })]).toEqual(['weather']);
78+
});
79+
80+
it('clears a single thread and all threads', () => {
81+
const store = new LegacyMapLoadedToolStore({ ttl: 0 });
82+
store.addLoaded(['weather'], { threadId: 'thread-1', args: emptyArgs });
83+
store.addLoaded(['calendar'], { threadId: 'thread-2', args: emptyArgs });
84+
85+
store.clearState('thread-1');
86+
expect(store.getLoadedNames({ threadId: 'thread-1', args: emptyArgs }).size).toBe(0);
87+
expect([...store.getLoadedNames({ threadId: 'thread-2', args: emptyArgs })]).toEqual(['calendar']);
88+
89+
store.clearAllState();
90+
expect(store.getLoadedNames({ threadId: 'thread-2', args: emptyArgs }).size).toBe(0);
91+
});
92+
93+
it('evicts stale state past the ttl and reports stats', async () => {
94+
const store = new LegacyMapLoadedToolStore({ ttl: 40 });
95+
store.addLoaded(['weather'], { threadId: 'thread-1', args: emptyArgs });
96+
expect(store.getStateStats().threadCount).toBe(1);
97+
98+
await new Promise(r => setTimeout(r, 70));
99+
expect(store.cleanupStaleState()).toBeGreaterThanOrEqual(1);
100+
expect(store.getStateStats().threadCount).toBe(0);
101+
});
102+
});
103+
104+
describe('ContextLoadedToolStore', () => {
105+
it('derives loaded names purely from the messages (restart-safe)', () => {
106+
const store = new ContextLoadedToolStore();
107+
const args = argsWithMessages([{ toolName: 'load_tool', result: { loaded: ['weather'] } }]);
108+
109+
// A brand-new store instance (simulating a process restart) still resolves
110+
// loaded names from the conversation messages alone.
111+
const names = store.getLoadedNames({ threadId: 'thread-1', args });
112+
expect([...names]).toEqual(['weather']);
113+
});
114+
115+
it('bridges activation with a same-process supplemental set before the messages catch up', () => {
116+
const store = new ContextLoadedToolStore();
117+
const emptyArgs = argsWithMessages([]);
118+
119+
store.addLoaded(['weather'], { threadId: 'thread-1', args: emptyArgs });
120+
// Messages do not yet contain the result, but the supplemental set carries it.
121+
expect([...store.getLoadedNames({ threadId: 'thread-1', args: emptyArgs })]).toEqual(['weather']);
122+
});
123+
124+
it('hands ownership to the messages once the result appears, so eviction de-loads', () => {
125+
const store = new ContextLoadedToolStore();
126+
const withResult = argsWithMessages([{ toolName: 'load_tool', result: { loaded: ['weather'] } }]);
127+
128+
store.addLoaded(['weather'], { threadId: 'thread-1', args: withResult });
129+
// First read sees it in the messages and prunes the supplemental entry.
130+
expect([...store.getLoadedNames({ threadId: 'thread-1', args: withResult })]).toEqual(['weather']);
131+
132+
// Simulate the result block leaving the messages.
133+
const evicted = argsWithMessages([]);
134+
expect(store.getLoadedNames({ threadId: 'thread-1', args: evicted }).size).toBe(0);
135+
});
136+
137+
it('does not share supplemental state across anonymous (no-threadId) requests', () => {
138+
const store = new ContextLoadedToolStore();
139+
const emptyArgs = argsWithMessages([]);
140+
141+
store.addLoaded(['weather'], { threadId: undefined, args: emptyArgs });
142+
expect(store.getLoadedNames({ threadId: undefined, args: emptyArgs }).size).toBe(0);
143+
});
144+
145+
it('drops the supplemental entry once it empties, so thread keys do not leak', () => {
146+
const store = new ContextLoadedToolStore();
147+
const supplemental = (store as unknown as { supplemental: Map<string, Set<string>> }).supplemental;
148+
149+
// Activated before the messages catch up -> entry created.
150+
store.addLoaded(['weather'], { threadId: 'thread-1', args: argsWithMessages([]) });
151+
expect(supplemental.has('thread-1')).toBe(true);
152+
153+
// Once the result appears in the messages, the name is pruned and the now-empty
154+
// entry is removed from the map rather than lingering as a dead key.
155+
const withResult = argsWithMessages([{ toolName: 'load_tool', result: { loaded: ['weather'] } }]);
156+
store.getLoadedNames({ threadId: 'thread-1', args: withResult });
157+
expect(supplemental.has('thread-1')).toBe(false);
158+
});
159+
});

0 commit comments

Comments
 (0)