Skip to content

Commit a763592

Browse files
devin-ai-integration[bot]TylerBarnesMastra Code (openai/gpt-5.5)
authored
feat: add multimodal content support for MCP tools and auto-detection in tool results (#16866)
## Description Adds native multimodal tool-result support for MCP-style `{ content: [...] }` arrays without duplicating image/audio payloads into `providerMetadata.mastra.modelOutput`. This lets MCP tools like the CUA driver return screenshots, window snapshots, and zoomed crops that are sent to the model as real image/file parts while stored messages keep the raw tool result. **Changes:** 1. **MCP client** (`packages/mcp/src/client/client.ts`, `packages/mcp/src/client/client.test.ts`): MCP tools no longer attach a generated `toModelOutput`, avoiding duplicated base64 payloads in persisted `providerMetadata`. Tests lock in that MCP tools rely on the shared tool-result conversion path instead. 2. **Core tool-result conversion** (`packages/core/src/agent/message-list/conversion/output-converter.ts`): During AIV5 UI → model message conversion, Mastra collects raw tool results from stored messages and converts MCP `content` arrays into provider-native `tool-result` content (`text`, `image-data`, and `file-data`). Explicit tool outputs and stored `providerMetadata.mastra.modelOutput` still take precedence, malformed media parts are preserved as text, and unknown MCP content parts are serialized as text instead of being dropped. 3. **Observational memory token counting** (`packages/memory/src/processors/observational-memory/token-counter.ts`): MCP multimodal tool-result parts are counted as attachment-like content instead of base64 JSON. Client-supplied `providerMetadata.mastra.tokenEstimate` values are honored per part, image/file heuristics are used as fallback, and multimodal fingerprints now use SHA-256. 4. **Tests and release notes**: Added coverage for MCP multimodal prompt conversion, explicit output precedence, malformed/unknown content preservation, stored `modelOutput` precedence, and token counting with client estimates. The changeset documents the new MCP-shaped multimodal tool-result support. **How it works:** - MCP tool returns raw content: `{ content: [{ type: 'text', text: '...' }, { type: 'image', data: 'base64...', mimeType: 'image/png' }] }` - Mastra stores that raw result on the tool invocation without adding a duplicate `modelOutput` payload. - When building the LLM prompt, the message conversion pipeline rewrites the matching default tool-result output to: `{ type: 'content', value: [{ type: 'text', text: '...' }, { type: 'image-data', data: 'base64...', mimeType: 'image/png' }] }` - Observational memory counts those multimodal content parts using client estimates when present, otherwise the normal image/file token heuristics. **Manual validation:** - Verified end-to-end with the CUA MCP server using `screenshot`, `get_window_state`, and `zoom` image-returning tools. - Verified CUA screenshots can be read from Safari/Calculator/iPhone Mirroring tool results without relying on MCP `toModelOutput` duplication. ## Related issue(s) Enables background computer use via CUA MCP server (discussion in #mastracode Slack thread). ## Type of change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have addressed all Coderabbit comments on this PR Link to Devin session: https://app.devin.ai/sessions/9df089dee5df4234a6c43cadc5f3dd4b Requested by: @TylerBarnes <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16866?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: tyler <tylerdbarnes@gmail.com> Co-authored-by: Mastra Code (openai/gpt-5.5) <noreply@mastra.ai>
1 parent 59dde92 commit a763592

9 files changed

Lines changed: 633 additions & 11 deletions

File tree

‎.changeset/thick-loops-wear.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
'@mastra/mcp': minor
3+
'@mastra/core': patch
4+
---
5+
6+
Added native multimodal tool-result support. Core now converts MCP-style tool results with image and audio `content` parts into model-native media output when building model prompts, without requiring MCP tools to persist duplicate media payloads in `providerMetadata.mastra.modelOutput`.
7+
8+
```ts
9+
return {
10+
content: [
11+
{ type: 'text', text: 'Screenshot captured' },
12+
{ type: 'image', data: base64Png, mimeType: 'image/png' },
13+
],
14+
};
15+
```

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@
8989
"prettier:format": "prettier --write . --log-level warn",
9090
"prettier:format:check": "prettier --check .",
9191
"list-changed-files": "(git diff --name-only --diff-filter=ACMR $(git symbolic-ref refs/remotes/origin/HEAD 2>/dev/null | sed 's@^refs/remotes/origin/@@' || echo 'main')...HEAD && git diff --name-only --diff-filter=ACMR && git diff --cached --name-only --diff-filter=ACMR) | sort -u",
92-
"list-prettier-files": "pnpm --silent list-changed-files | grep -E '\\.(js|mjs|ts|tsx|md|mdx|json|graphql|gql)$' | while IFS= read -r file; do [ -L \"$file\" ] || printf '%s\\n' \"$file\"; done || true",
92+
"list-prettier-files": "pnpm --silent list-changed-files | grep -E '\\.(js|mjs|ts|tsx|md|mdx|json|graphql|gql)$' | while IFS= read -r file; do [ ! -f \"$file\" ] || [ -L \"$file\" ] || printf '%s\\n' \"$file\"; done || true",
9393
"prettier:changed": "pnpm --silent list-prettier-files | xargs prettier --write --log-level warn",
9494
"typecheck": "pnpm --filter \"!./explorations/**/*\" -r typecheck",
9595
"format": "pnpm turbo --filter \"!./examples/**/*\" --filter \"!./docs/**/*\" --filter \"!@internal/playground\" lint -- --fix",

‎packages/cli/src/utils/source-hash.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ async function writeFileSynced(path: string, content: string): Promise<void> {
2323
}
2424
}
2525

26-
describe('source-hash', () => {
26+
describe.sequential('source-hash', () => {
2727
let testDir: string;
2828
let mastraDir: string;
2929
let outputDir: string;

‎packages/core/src/agent/message-list/conversion/output-converter-file-metadata.test.ts‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,3 +187,59 @@ describe('aiV5UIMessagesToAIV5ModelMessages — assistant file providerMetadata'
187187
expect(fileParts[1].providerOptions).toEqual({ google: { thoughtSignature: 'sig-2' } });
188188
});
189189
});
190+
191+
describe('aiV5UIMessagesToAIV5ModelMessages — MCP content tool result output', () => {
192+
it('does not override tool outputs that already differ from the raw stored result', () => {
193+
const rawOutput = {
194+
content: [
195+
{ type: 'text', text: 'raw text' },
196+
{ type: 'image', data: 'base64data', mimeType: 'image/png' },
197+
],
198+
};
199+
200+
const messages: AIV5Type.UIMessage[] = [
201+
{
202+
id: 'msg-tool-ui',
203+
role: 'assistant',
204+
parts: [
205+
{
206+
type: 'tool-screenshot',
207+
toolCallId: 'call-mcp-image',
208+
state: 'output-available',
209+
input: {},
210+
output: { type: 'text', value: 'Explicit summary wins' },
211+
} as any,
212+
],
213+
},
214+
];
215+
216+
const dbMessages = [
217+
{
218+
id: 'msg-tool-db',
219+
role: 'assistant',
220+
createdAt: new Date(),
221+
content: {
222+
format: 2,
223+
parts: [
224+
{
225+
type: 'tool-invocation',
226+
toolInvocation: {
227+
toolCallId: 'call-mcp-image',
228+
toolName: 'screenshot',
229+
state: 'result',
230+
args: {},
231+
result: rawOutput,
232+
},
233+
},
234+
],
235+
},
236+
},
237+
];
238+
239+
const result = aiV5UIMessagesToAIV5ModelMessages(messages, dbMessages as any);
240+
const toolMessage = result.find(message => message.role === 'tool');
241+
const toolResult = (toolMessage!.content as any[]).find(part => part.type === 'tool-result');
242+
243+
expect(toolResult.output).toEqual({ type: 'text', value: 'Explicit summary wins' });
244+
});
245+
});

‎packages/core/src/agent/message-list/conversion/output-converter.ts‎

Lines changed: 88 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,92 @@ export function aiV4UIMessagesToAIV4CoreMessages(messages: UIMessageV4[]): CoreM
285285
return convertToCoreMessagesV4(sanitizeAIV4UIMessages(messages));
286286
}
287287

288+
/**
289+
* Converts MCP-style tool results (`{ content: [...] }`) to model-native
290+
* multimodal tool result output without persisting a duplicate modelOutput copy.
291+
*/
292+
function convertMcpContentToolResultOutput(output: unknown): unknown {
293+
if (!output || typeof output !== 'object') return undefined;
294+
295+
const content = (output as Record<string, unknown>).content;
296+
if (!Array.isArray(content)) return undefined;
297+
298+
const hasValidMultimodal = content.some(part => {
299+
if (!part || typeof part !== 'object') return false;
300+
const typedPart = part as Record<string, unknown>;
301+
return (typedPart.type === 'image' || typedPart.type === 'audio') && typeof typedPart.data === 'string';
302+
});
303+
if (!hasValidMultimodal) return undefined;
304+
305+
const value = content
306+
.map(part => {
307+
if (!part || typeof part !== 'object') return null;
308+
const typedPart = part as Record<string, unknown>;
309+
switch (typedPart.type) {
310+
case 'text':
311+
return { type: 'text', text: String(typedPart.text ?? '') };
312+
case 'image':
313+
return typeof typedPart.data === 'string'
314+
? { type: 'image-data', data: typedPart.data, mediaType: String(typedPart.mimeType ?? 'image/png') }
315+
: { type: 'text', text: JSON.stringify(typedPart) };
316+
case 'audio':
317+
return typeof typedPart.data === 'string'
318+
? { type: 'file-data', data: typedPart.data, mediaType: String(typedPart.mimeType ?? 'audio/wav') }
319+
: { type: 'text', text: JSON.stringify(typedPart) };
320+
default:
321+
return { type: 'text', text: JSON.stringify(typedPart) };
322+
}
323+
})
324+
.filter(Boolean);
325+
326+
return value.length > 0 ? { type: 'content', value } : undefined;
327+
}
328+
329+
function collectRawToolResultOutputs(dbMessages: MastraDBMessage[]): Map<string, unknown> {
330+
const outputs = new Map<string, unknown>();
331+
for (const message of dbMessages) {
332+
if (message.content?.format !== 2 || !message.content.parts) continue;
333+
334+
for (const part of message.content.parts) {
335+
if (part.type !== 'tool-invocation' || part.toolInvocation?.state !== 'result') continue;
336+
outputs.set(part.toolInvocation.toolCallId, part.toolInvocation.result);
337+
}
338+
}
339+
return outputs;
340+
}
341+
342+
function isDefaultToolResultOutput(output: unknown, rawOutput: unknown): boolean {
343+
if (!output || typeof output !== 'object') return false;
344+
const typedOutput = output as Record<string, unknown>;
345+
if (typedOutput.type !== 'json') return false;
346+
return JSON.stringify(typedOutput.value) === JSON.stringify(rawOutput);
347+
}
348+
349+
function applyMcpContentToolResultOutputs(
350+
modelMessages: AIV5Type.ModelMessage[],
351+
dbMessages: MastraDBMessage[],
352+
): AIV5Type.ModelMessage[] {
353+
const rawOutputs = collectRawToolResultOutputs(dbMessages);
354+
if (rawOutputs.size === 0) return modelMessages;
355+
356+
return modelMessages.map(message => {
357+
if (message.role !== 'tool' || !Array.isArray(message.content)) return message;
358+
359+
let modified = false;
360+
const content = message.content.map(part => {
361+
if (part.type !== 'tool-result' || !rawOutputs.has(part.toolCallId)) return part;
362+
const rawOutput = rawOutputs.get(part.toolCallId);
363+
if (!isDefaultToolResultOutput(part.output, rawOutput)) return part;
364+
const converted = convertMcpContentToolResultOutput(rawOutput);
365+
if (!converted) return part;
366+
modified = true;
367+
return { ...part, output: converted } as typeof part;
368+
});
369+
370+
return modified ? ({ ...message, content } as AIV5Type.ModelMessage) : message;
371+
});
372+
}
373+
288374
/**
289375
* Restores `providerOptions` on assistant file parts after `convertToModelMessages`.
290376
*
@@ -380,9 +466,10 @@ export function aiV5UIMessagesToAIV5ModelMessages(
380466
}
381467

382468
const withFileMetadata = restoreAssistantFileProviderMetadata(converted, preprocessed);
469+
const withMcpContentOutputs = applyMcpContentToolResultOutputs(withFileMetadata, dbMessages);
383470

384471
// Add input field to tool-result parts for Anthropic API compatibility (fixes issue #11376)
385-
const anthropicCompat = ensureAnthropicCompatibleMessages(withFileMetadata, dbMessages);
472+
const anthropicCompat = ensureAnthropicCompatibleMessages(withMcpContentOutputs, dbMessages);
386473

387474
return filterIncompleteToolCalls ? sanitizeOrphanedToolPairs(anthropicCompat) : anthropicCompat;
388475
}

‎packages/core/src/agent/message-list/tests/message-list-v5.test.ts‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1841,6 +1841,151 @@ describe('MessageList V5 Support', () => {
18411841
expect(toolPart.toolInvocation.result).toEqual({ status: 200, body: 'lots of data here' });
18421842
});
18431843

1844+
it('should convert MCP content-array tool results to multimodal model output without providerMetadata duplication', async () => {
1845+
const list = new MessageList({ threadId, resourceId });
1846+
1847+
list.add('Take a screenshot', 'input');
1848+
1849+
const toolResultMessage: MastraDBMessage = {
1850+
id: 'msg-mcp-image-tool',
1851+
role: 'assistant',
1852+
createdAt: new Date(),
1853+
threadId,
1854+
resourceId,
1855+
content: {
1856+
format: 2,
1857+
parts: [
1858+
{
1859+
type: 'tool-invocation',
1860+
toolInvocation: {
1861+
toolCallId: 'call-mcp-image',
1862+
toolName: 'screenshot',
1863+
state: 'result',
1864+
args: {},
1865+
result: {
1866+
content: [
1867+
{ type: 'text', text: 'Screenshot captured' },
1868+
{ type: 'image', data: 'base64data', mimeType: 'image/png' },
1869+
{ type: 'audio', mimeType: 'audio/wav' },
1870+
{ type: 'resource', resource: { uri: 'file:///tmp/output.txt', text: 'details' } },
1871+
],
1872+
},
1873+
},
1874+
},
1875+
],
1876+
},
1877+
};
1878+
1879+
list.add(toolResultMessage, 'response');
1880+
1881+
const dbMessages = list.get.all.db();
1882+
const toolDbMsg = dbMessages.find(m => m.id === 'msg-mcp-image-tool');
1883+
const toolPart = toolDbMsg?.content.parts?.[0] as any;
1884+
expect(toolPart.providerMetadata?.mastra?.modelOutput).toBeUndefined();
1885+
expect(toolPart.toolInvocation.result.content[1].data).toBe('base64data');
1886+
1887+
const prompt = await list.get.all.aiV5.llmPrompt();
1888+
const toolRole = prompt.find(m => m.role === 'tool');
1889+
const toolResultPart = (toolRole as any).content.find((p: any) => p.type === 'tool-result');
1890+
expect(toolResultPart.output).toEqual({
1891+
type: 'content',
1892+
value: [
1893+
{ type: 'text', text: 'Screenshot captured' },
1894+
{ type: 'image-data', data: 'base64data', mediaType: 'image/png' },
1895+
{ type: 'text', text: JSON.stringify({ type: 'audio', mimeType: 'audio/wav' }) },
1896+
{
1897+
type: 'text',
1898+
text: JSON.stringify({ type: 'resource', resource: { uri: 'file:///tmp/output.txt', text: 'details' } }),
1899+
},
1900+
],
1901+
});
1902+
});
1903+
1904+
it('should preserve explicit modelOutput over MCP-style raw content in llmPrompt', async () => {
1905+
const list = new MessageList({ threadId, resourceId });
1906+
1907+
list.add('Summarize tool output', 'input');
1908+
list.add(
1909+
{
1910+
id: 'msg-explicit-model-output-mcp-shape',
1911+
role: 'assistant',
1912+
createdAt: new Date(),
1913+
threadId,
1914+
resourceId,
1915+
content: {
1916+
format: 2,
1917+
parts: [
1918+
{
1919+
type: 'tool-invocation',
1920+
toolInvocation: {
1921+
toolCallId: 'call-explicit-model-output-mcp-shape',
1922+
toolName: 'customTool',
1923+
state: 'result',
1924+
args: {},
1925+
result: {
1926+
content: [
1927+
{ type: 'text', text: 'raw text' },
1928+
{ type: 'image', data: 'raw-base64', mimeType: 'image/png' },
1929+
],
1930+
},
1931+
},
1932+
providerMetadata: {
1933+
mastra: {
1934+
modelOutput: { type: 'text', value: 'Explicit summary wins' },
1935+
},
1936+
},
1937+
},
1938+
],
1939+
},
1940+
},
1941+
'response',
1942+
);
1943+
1944+
const prompt = await list.get.all.aiV5.llmPrompt();
1945+
const toolRole = prompt.find(m => m.role === 'tool');
1946+
const toolResultPart = (toolRole as any).content.find((p: any) => p.type === 'tool-result');
1947+
expect(toolResultPart.output).toEqual({ type: 'text', value: 'Explicit summary wins' });
1948+
});
1949+
1950+
it('should leave malformed MCP multimodal content as a regular tool result', async () => {
1951+
const list = new MessageList({ threadId, resourceId });
1952+
1953+
list.add('Take a screenshot', 'input');
1954+
list.add(
1955+
{
1956+
id: 'msg-mcp-invalid-image-tool',
1957+
role: 'assistant',
1958+
createdAt: new Date(),
1959+
threadId,
1960+
resourceId,
1961+
content: {
1962+
format: 2,
1963+
parts: [
1964+
{
1965+
type: 'tool-invocation',
1966+
toolInvocation: {
1967+
toolCallId: 'call-mcp-invalid-image',
1968+
toolName: 'screenshot',
1969+
state: 'result',
1970+
args: {},
1971+
result: { content: [{ type: 'image', mimeType: 'image/png' }] },
1972+
},
1973+
},
1974+
],
1975+
},
1976+
},
1977+
'response',
1978+
);
1979+
1980+
const prompt = await list.get.all.aiV5.llmPrompt();
1981+
const toolRole = prompt.find(m => m.role === 'tool');
1982+
const toolResultPart = (toolRole as any).content.find((p: any) => p.type === 'tool-result');
1983+
expect(toolResultPart.output).toEqual({
1984+
type: 'json',
1985+
value: { content: [{ type: 'image', mimeType: 'image/png' }] },
1986+
});
1987+
});
1988+
18441989
it('should apply payload transforms to UI and drained transcript without mutating model messages', () => {
18451990
const list = new MessageList({ threadId, resourceId });
18461991

0 commit comments

Comments
 (0)