Skip to content

fix(openai-compatible): buffer tool call deltas until function.name arrives - #14760

Merged
aayush-kapoor merged 4 commits into
mainfrom
aayush/oai-compat-tool-streaming
Apr 27, 2026
Merged

aayush-kapoor merged 4 commits into
mainfrom
aayush/oai-compat-tool-streaming

Conversation

@aayush-kapoor

Copy link
Copy Markdown
Collaborator

Background

#14722

Some OpenAI-compatible providers send the first tool_calls delta with an id and function.arguments but without function.name, then provide function.name in a subsequent delta. The StreamingToolCallTracker previously threw AI_InvalidResponseDataError on the first delta because it required function.name to be present immediately.

Summary

added a small buffering layer in front of the tracker that:

  • holds onto deltas (per index) until function.name is seen
  • while buffering, it accumulates id, arguments, and extra_content
  • when function.name shows up, it forwards a single combined delta to StreamingToolCallTracker.processDelta
  • once forwarded, the tool-call index is marked as forwarded and all subsequent deltas (argument-only chunks) pass straight through to the tracker unchanged.

Manual Verification

verified by running this repro before and after the fix

repro:
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { streamText, tool } from 'ai';
import { z } from 'zod';
import { run } from '../../lib/run';

const encoder = new TextEncoder();

const chunks = [
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: { role: 'assistant', content: null },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            {
              index: 0,
              id: 'call_123',
              type: 'function',
              function: { arguments: '' },
            },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            {
              index: 0,
              id: 'call_123',
              type: 'function',
              function: { name: 'bash', arguments: '{' },
            },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            { index: 0, function: { arguments: '"command"' } },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            { index: 0, function: { arguments: ': "ls -la"}' } },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: 'tool_calls',
        index: 0,
        delta: {},
      },
    ],
  },
];

const openaiCompatible = createOpenAICompatible({
  name: 'late-tool-name-repro',
  baseURL: 'https://example.test/v1',
  apiKey: 'test-key',
  fetch: async () =>
    new Response(
      new ReadableStream({
        start(controller) {
          for (const chunk of chunks) {
            controller.enqueue(
              encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`),
            );
          }

          controller.enqueue(encoder.encode('data: [DONE]\n\n'));
          controller.close();
        },
      }),
      {
        headers: {
          'content-type': 'text/event-stream',
        },
      },
    ),
});

run(async () => {
  const result = streamText({
    model: openaiCompatible.chatModel('mock-model'),
    prompt: 'Run ls -la.',
    maxRetries: 0,
    toolChoice: 'required',
    tools: {
      bash: tool({
        description: 'Run a shell command.',
        inputSchema: z.object({
          command: z.string(),
        }),
        execute: async input => input,
      }),
    },
  });

  try {
    for await (const part of result.fullStream) {
      console.log(part.type, part);
    }
  } catch (error) {
    console.error('\nReproduced error:');
    console.error(error);
    process.exitCode = 1;
  }
});

Checklist

  • All commits are signed (PRs with unsigned commits cannot be merged)
  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Related Issues

fixes #14722

@aayush-kapoor
aayush-kapoor enabled auto-merge (squash) April 27, 2026 17:34
@aayush-kapoor
aayush-kapoor merged commit ab81968 into main Apr 27, 2026
17 checks passed
@aayush-kapoor
aayush-kapoor deleted the aayush/oai-compat-tool-streaming branch April 27, 2026 17:38
@aayush-kapoor aayush-kapoor added the backport Admins only: add this label to a pull request in order to backport it to the prior version label Apr 27, 2026
gr2m pushed a commit that referenced this pull request Jun 5, 2026
…rrives (#14760)

## Background

#14722

Some OpenAI-compatible providers send the first tool_calls delta with an
`id` and `function.arguments` but without `function.name`, then provide
function.name in a subsequent delta. The `StreamingToolCallTracker`
previously threw AI_InvalidResponseDataError on the first delta because
it required function.name to be present immediately.

## Summary

added a small buffering layer in front of the tracker that:
- holds onto deltas (per index) until function.name is seen
- while buffering, it accumulates id, arguments, and extra_content
- when function.name shows up, it forwards a single combined delta to
StreamingToolCallTracker.processDelta
- once forwarded, the tool-call index is marked as forwarded and all
subsequent deltas (argument-only chunks) pass straight through to the
tracker unchanged.

## Manual Verification

verified by running this repro before and after the fix

<details>
<summary>repro:</summary>

```ts
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { streamText, tool } from 'ai';
import { z } from 'zod';
import { run } from '../../lib/run';

const encoder = new TextEncoder();

const chunks = [
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: { role: 'assistant', content: null },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            {
              index: 0,
              id: 'call_123',
              type: 'function',
              function: { arguments: '' },
            },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            {
              index: 0,
              id: 'call_123',
              type: 'function',
              function: { name: 'bash', arguments: '{' },
            },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            { index: 0, function: { arguments: '"command"' } },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: null,
        index: 0,
        delta: {
          tool_calls: [
            { index: 0, function: { arguments: ': "ls -la"}' } },
          ],
        },
      },
    ],
  },
  {
    choices: [
      {
        finish_reason: 'tool_calls',
        index: 0,
        delta: {},
      },
    ],
  },
];

const openaiCompatible = createOpenAICompatible({
  name: 'late-tool-name-repro',
  baseURL: 'https://example.test/v1',
  apiKey: 'test-key',
  fetch: async () =>
    new Response(
      new ReadableStream({
        start(controller) {
          for (const chunk of chunks) {
            controller.enqueue(
              encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`),
            );
          }

          controller.enqueue(encoder.encode('data: [DONE]\n\n'));
          controller.close();
        },
      }),
      {
        headers: {
          'content-type': 'text/event-stream',
        },
      },
    ),
});

run(async () => {
  const result = streamText({
    model: openaiCompatible.chatModel('mock-model'),
    prompt: 'Run ls -la.',
    maxRetries: 0,
    toolChoice: 'required',
    tools: {
      bash: tool({
        description: 'Run a shell command.',
        inputSchema: z.object({
          command: z.string(),
        }),
        execute: async input => input,
      }),
    },
  });

  try {
    for await (const part of result.fullStream) {
      console.log(part.type, part);
    }
  } catch (error) {
    console.error('\nReproduced error:');
    console.error(error);
    process.exitCode = 1;
  }
});

```
</details>

## Checklist

- [x] All commits are signed (PRs with unsigned commits cannot be
merged)
- [x] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Related Issues

fixes #14722
gr2m added a commit that referenced this pull request Jul 6, 2026
….name arrives (#16836)

Manual backport of #14760 to `release-v6.0`, ported on top of the #16800
security backport since both rewrite the same delta-handling region. v6
has no `StreamingToolCallTracker`, so the buffering is implemented
inline: indexed tool-call deltas are buffered until the delta carrying
`function.name` arrives; deltas that never receive a name still fail
with the original `InvalidResponseDataError` during flush. Original
regression tests apply cleanly. openai-compatible suite passes locally
(210/210) plus all dependent provider packages (cerebras, deepseek,
groq, alibaba, fireworks, deepinfra, togetherai, baseten, huggingface,
moonshotai, vercel).

Part of the v6.0 backport tracking issue #16767.
gr2m added a commit that referenced this pull request Sep 30, 2026
…rrives (port to release-v5.0)

Replace the committed cherry-pick conflict markers with a port of #14760 that
fits release-v5.0, where streaming tool calls go through the shared
StreamingToolCallTracker: buffer index-keyed deltas in front of the tracker
until the delta carrying function.name arrives; deltas that never receive a
name still fail with InvalidResponseDataError on flush.

Co-Authored-By: Claude <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — be91e302 Deployed Apr 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Admins only: add this label to a pull request in order to backport it to the prior version task-merge-tracking-done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI-compatible streaming tool calls fail when function.name arrives in a later chunk

2 participants