fix(core): sanitize LLM tokens from tool-call input before JSON.parse (#13400)

## Summary

- Strip LLM-appended tokens (`<|call|>`, `<|endoftext|>`, etc.) from
tool-call input strings before `JSON.parse`, recovering valid arguments
that were previously silently discarded
- Add comprehensive tests verifying data recovery for real-world
malformed inputs reported in #13185 and #13261

## Problem

`convertFullStreamChunkToMastra()` in
`packages/core/src/stream/aisdk/v5/transform.ts` parses tool-call input
via `JSON.parse(value.input)`. The existing `try/catch` handles parse
failures gracefully (no crash), but **silently sets `toolCallInput =
undefined`** — meaning the tool call loses its arguments entirely.

In practice, many of these failures are recoverable:

| Source | Example Input | Root Cause |
|--------|--------------|------------|
| OpenAI models (#13185) | `'{"checkpointNumber": 1}\t<\|call\|>'` | LLM
appends internal `<\|call\|>` token after valid JSON |
| OpenAI models (#13185) | `'{}<\|call\|>'` | Empty args with trailing
token |
| OpenRouter/Novita (#13261) | `'{"checkpointNumber":?}'` | Provider
returns truly malformed JSON (not recoverable) |

The first two cases contain **valid JSON buried under LLM noise** —
stripping the tokens before parsing recovers the data.

## Solution

Added an exported `sanitizeToolCallInput(input: string)` function that
removes `<|...|>` token patterns and surrounding whitespace via regex
before the input reaches `JSON.parse`:

```typescript
export function sanitizeToolCallInput(input: string): string {
  return input.replace(/[\s]*<\|[^|]*\|>[\s]*/g, '').trim();
}
```

The `tool-call` case now sanitizes first, then parses:

```typescript
const sanitized = sanitizeToolCallInput(value.input);
try {
  toolCallInput = JSON.parse(sanitized);
} catch (error) {
  // ... existing error handling unchanged
}
```

## What Changed

| File | Change |
|------|--------|
| `packages/core/src/stream/aisdk/v5/transform.ts` | Added
`sanitizeToolCallInput()`, updated `tool-call` case to sanitize before
parse |
| `packages/core/src/stream/aisdk/v5/transform.test.ts` | Added 12 new
tests: 4 integration (data recovery) + 8 unit (sanitizer) |

## Tests

**New integration tests** verify actual data recovery (not just
no-throw):
- ✅ `'{}<|call|>'` → recovers `{}`
- ✅ `'{"checkpointNumber": 1, ...}\t<|call|>'` → recovers full object
- ✅ `'{"query": "hello world"}<|endoftext|>'` → recovers object
- ✅ `'{"checkpointNumber":?}'` → gracefully returns `undefined`

**New unit tests** for `sanitizeToolCallInput`:
- ✅ Strips `<|call|>`, `<|endoftext|>`, `<|end|>` tokens
- ✅ Strips multiple consecutive tokens
- ✅ Strips tab + token combinations
- ✅ No-op on clean JSON
- ✅ Handles empty string

All 21 tests pass. Build succeeds. Zero lint/type errors.

Closes #13185
Closes #13261

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved resilience of streamed tool-call input handling by stripping
internal formatting tokens and surrounding whitespace so valid JSON
payloads are preserved and parsing failures are reduced.

* **Tests**
* Added comprehensive tests for malformed/partial JSON,
trailing/token/whitespace variations, and preservation of token-like
text inside string values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Roamin <97863888+roaminro@users.noreply.github.com>

Author: PPRAMANIK62

.changeset/sanitize-tool-call-json-input.md

@@ -0,0 +1,11 @@
+---
+"@mastra/core": patch
+---
+
+**Fixed tool-call arguments being silently lost when LLMs append internal tokens to JSON**
+
+LLMs (particularly via OpenRouter and OpenAI) sometimes append internal tokens like `<|call|>`, `<|endoftext|>`, or `<|end|>` to otherwise valid JSON in streamed tool-call arguments. Previously, these inputs would fail `JSON.parse` and the tool call would silently lose its arguments (set to `undefined`).
+
+Now, `sanitizeToolCallInput` strips these token patterns before parsing, recovering valid data that was previously discarded. Valid JSON containing `<|...|>` inside string values is left untouched. Truly malformed JSON still gracefully returns `undefined`.
+
+Fixes https://github.com/mastra-ai/mastra/issues/13185 and https://github.com/mastra-ai/mastra/issues/13261.

e2e-tests/create-mastra/create-mastra.test.ts

@@ -126,7 +126,7 @@ describe('create mastra', () => {
 
                 Use the weatherTool to fetch current weather data.
           ",
-              "modelId": "gpt-4o",
+              "modelId": "gpt-5-mini",
               "modelVersion": "v2",
               "name": "Weather Agent",
               "outputProcessors": [],

packages/core/src/stream/aisdk/v5/transform.test.ts

@@ -1,6 +1,6 @@
-import { describe, it, expect } from 'vitest';
+import { describe, it, expect, vi } from 'vitest';
 import { ChunkFrom } from '../../types';
-import { convertFullStreamChunkToMastra } from './transform';
+import { convertFullStreamChunkToMastra, sanitizeToolCallInput } from './transform';
 import type { StreamPart } from './transform';
 
 describe('convertFullStreamChunkToMastra', () => {
@@ -180,6 +180,160 @@ describe('convertFullStreamChunkToMastra', () => {
         throw new Error('Result is not a tool-call');
       }
     });
+
+    it('should recover valid JSON with trailing <|call|> token', () => {
+      const chunk: StreamPart = {
+        type: 'tool-call',
+        toolCallId: 'call-1',
+        toolName: 'get_weather',
+        input: '{}<|call|>',
+        providerExecuted: false,
+      };
+
+      const result = convertFullStreamChunkToMastra(chunk, { runId: 'test-run-123' });
+
+      expect(result).toBeDefined();
+      expect(result?.type).toBe('tool-call');
+      if (result?.type === 'tool-call') {
+        expect(result.payload.args).toEqual({});
+      }
+    });
+
+    it('should recover valid JSON with tab + <|call|> token (issue #13185)', () => {
+      const chunk: StreamPart = {
+        type: 'tool-call',
+        toolCallId: 'call-2',
+        toolName: 'checkpoint',
+        input: '{\n"checkpointNumber": 1,\n"vehicleType": "leopard"\n}\t<|call|>',
+        providerExecuted: false,
+      };
+
+      const result = convertFullStreamChunkToMastra(chunk, { runId: 'test-run-123' });
+
+      expect(result).toBeDefined();
+      expect(result?.type).toBe('tool-call');
+      if (result?.type === 'tool-call') {
+        expect(result.payload.args).toEqual({ checkpointNumber: 1, vehicleType: 'leopard' });
+      }
+    });
+
+    it('should recover valid JSON with <|endoftext|> token', () => {
+      const chunk: StreamPart = {
+        type: 'tool-call',
+        toolCallId: 'call-3',
+        toolName: 'search',
+        input: '{"query": "hello world"}<|endoftext|>',
+        providerExecuted: false,
+      };
+
+      const result = convertFullStreamChunkToMastra(chunk, { runId: 'test-run-123' });
+
+      expect(result).toBeDefined();
+      expect(result?.type).toBe('tool-call');
+      if (result?.type === 'tool-call') {
+        expect(result.payload.args).toEqual({ query: 'hello world' });
+      }
+    });
+
+    it('should gracefully return undefined for truly malformed JSON (issue #13261)', () => {
+      const chunk: StreamPart = {
+        type: 'tool-call',
+        toolCallId: 'call-4',
+        toolName: 'checkpoint',
+        input: '{"vehicleType":"leopard","checkpointNumber":?}',
+        providerExecuted: false,
+      };
+
+      const result = convertFullStreamChunkToMastra(chunk, { runId: 'test-run-123' });
+
+      expect(result).toBeDefined();
+      expect(result?.type).toBe('tool-call');
+      if (result?.type === 'tool-call') {
+        expect(result.payload.args).toBeUndefined();
+      }
+    });
+
+    it('should preserve <|...|> patterns inside JSON string values in tool-call args', () => {
+      const chunk: StreamPart = {
+        type: 'tool-call',
+        toolCallId: 'call-5',
+        toolName: 'process_text',
+        input: '{"text": "The <|endoftext|> token marks boundaries"}',
+        providerExecuted: false,
+      };
+
+      const result = convertFullStreamChunkToMastra(chunk, { runId: 'test-run-123' });
+
+      expect(result).toBeDefined();
+      expect(result?.type).toBe('tool-call');
+      if (result?.type === 'tool-call') {
+        expect(result.payload.args).toEqual({ text: 'The <|endoftext|> token marks boundaries' });
+      }
+    });
+
+    it('should return undefined args without console.error noise when input is purely LLM tokens', () => {
+      const errorSpy = vi.spyOn(console, 'error');
+      const chunk: StreamPart = {
+        type: 'tool-call',
+        toolCallId: 'call-6',
+        toolName: 'noop',
+        input: '<|call|>',
+        providerExecuted: false,
+      };
+
+      const result = convertFullStreamChunkToMastra(chunk, { runId: 'test-run-123' });
+      expect(result).toBeDefined();
+      expect(result?.type).toBe('tool-call');
+      if (result?.type === 'tool-call') {
+        expect(result.payload.args).toBeUndefined();
+      }
+      expect(errorSpy).not.toHaveBeenCalled();
+      errorSpy.mockRestore();
+    });
+  });
+
+  describe('sanitizeToolCallInput', () => {
+    it('should strip <|call|> token from valid JSON', () => {
+      expect(sanitizeToolCallInput('{}<|call|>')).toBe('{}');
+    });
+
+    it('should strip <|endoftext|> token', () => {
+      expect(sanitizeToolCallInput('{"a":1}<|endoftext|>')).toBe('{"a":1}');
+    });
+
+    it('should strip multiple tokens', () => {
+      expect(sanitizeToolCallInput('{}<|call|><|endoftext|>')).toBe('{}');
+    });
+
+    it('should strip tab + token combinations', () => {
+      expect(sanitizeToolCallInput('{}\t<|call|>')).toBe('{}');
+    });
+
+    it('should be a no-op on clean JSON', () => {
+      expect(sanitizeToolCallInput('{"key": "value"}')).toBe('{"key": "value"}');
+    });
+
+    it('should handle empty string', () => {
+      expect(sanitizeToolCallInput('')).toBe('');
+    });
+
+    it('should strip <|end|> token', () => {
+      expect(sanitizeToolCallInput('{"x":1}<|end|>')).toBe('{"x":1}');
+    });
+
+    it('should strip tokens with surrounding whitespace', () => {
+      expect(sanitizeToolCallInput('{"x":1}  <|call|>  ')).toBe('{"x":1}');
+    });
+
+    it('should preserve <|...|> patterns inside JSON string values', () => {
+      const input = '{"text": "use <|call|> token"}';
+      expect(sanitizeToolCallInput(input)).toBe('{"text": "use <|call|> token"}');
+    });
+
+    it('should preserve multiple <|...|> patterns inside JSON string values', () => {
+      const input = '{"prompt": "tokens: <|endoftext|> and <|call|> are special"}';
+      expect(sanitizeToolCallInput(input)).toBe('{"prompt": "tokens: <|endoftext|> and <|call|> are special"}');
+    });
   });
 
   describe('other chunk types', () => {

packages/core/src/stream/aisdk/v5/transform.ts

@@ -11,6 +11,34 @@ import type { ChunkType, LanguageModelUsage } from '../../types';
 import { ChunkFrom } from '../../types';
 import { DefaultGeneratedFile, DefaultGeneratedFileWithType } from './file';
 
+/**
+ * Sanitizes tool-call input strings for safe JSON parsing.
+ *
+ * LLMs sometimes append internal tokens like `<|call|>`, `<|endoftext|>`, or `<|end|>`
+ * to otherwise valid JSON in streamed tool-call arguments, causing JSON.parse to fail.
+ *
+ * This function first attempts JSON.parse on the original input. If parsing succeeds,
+ * the original string is returned unchanged — this avoids corrupting valid JSON payloads
+ * that legitimately contain `<|...|>` patterns inside string values.
+ *
+ * Only when the original input is not valid JSON does the function fall back to stripping
+ * `<|...|>` token patterns and surrounding whitespace via regex.
+ *
+ * @see https://github.com/mastra-ai/mastra/issues/13261
+ * @see https://github.com/mastra-ai/mastra/issues/13185
+ */
+export function sanitizeToolCallInput(input: string): string {
+  // Fast path: if input is already valid JSON, return unchanged to avoid
+  // corrupting <|...|> patterns that appear inside JSON string values.
+  try {
+    JSON.parse(input);
+    return input;
+  } catch {
+    // Input is not valid JSON — strip LLM-specific tokens and retry
+    return input.replace(/[\s]*<\|[^|]*\|>[\s]*/g, '').trim();
+  }
+}
+
 export type StreamPart =
   | Exclude<LanguageModelV2StreamPart, { type: 'finish' }>
   | {
@@ -134,14 +162,17 @@ export function convertFullStreamChunkToMastra(value: StreamPart, ctx: { runId:
       let toolCallInput: Record<string, any> | undefined = undefined;
 
       if (value.input) {
-        try {
-          toolCallInput = JSON.parse(value.input);
-        } catch (error) {
-          console.error('Error converting tool call input to JSON', {
-            error,
-            input: value.input,
-          });
-          toolCallInput = undefined;
+        const sanitized = sanitizeToolCallInput(value.input);
+        if (sanitized) {
+          try {
+            toolCallInput = JSON.parse(sanitized);
+          } catch (error) {
+            console.error('Error converting tool call input to JSON', {
+              error,
+              input: value.input,
+            });
+            toolCallInput = undefined;
+          }
         }
       }