Skip to content

Conversation

@ibetitsmike
Copy link
Contributor

@ibetitsmike ibetitsmike commented Jan 10, 2026

What

Rewrite provider requests so tool-result media (e.g. Chrome DevTools take_screenshot) is not sent as large base64 blobs inside tool-result JSON.

Concretely:

  • Replace tool-result { type: "media", data: "<base64>", mediaType } entries with small text placeholders.
  • Emit a synthetic user message right after the assistant/tool message that attaches the extracted image(s) as multimodal parts.

This is applied both:

  • When building the initial request from persisted history (AIService), and
  • For internal streamText() steps via prepareStep, so tool results produced mid-stream are also rewritten before the next LLM call.

Why

Some providers treat tool-result payloads as text/JSON, so embedding screenshot base64 in tool results can easily exceed context limits ("prompt too long"). Attaching screenshots as proper image/file parts avoids that failure mode.

Notes

This is request-only: persisted history/tool output remains unchanged for UI rendering.

Validation

  • Unit tests for the request transforms
  • make static-check

Generated with mux • Model: openai:gpt-5.2 • Thinking: xhigh • Cost: 3119344{cost}

@ibetitsmike ibetitsmike changed the title 🤖 fix: prevent screenshot base64 from dominating token stats 🤖 fix: avoid screenshot base64 blobs in token stats and requests Jan 12, 2026
Avoid sending base64 screenshots inside tool-result JSON by extracting media tool outputs into synthetic user file parts during request construction.
@ibetitsmike ibetitsmike force-pushed the mike/fix-mcp-screenshot-token-count branch from 3ce5d2f to 6252c39 Compare January 12, 2026 14:03
@ibetitsmike ibetitsmike changed the title 🤖 fix: avoid screenshot base64 blobs in token stats and requests 🤖 fix: attach tool screenshots as image parts Jan 12, 2026
@ibetitsmike ibetitsmike added this pull request to the merge queue Jan 12, 2026
Merged via the queue into main with commit 98a6ee0 Jan 12, 2026
21 checks passed
@ibetitsmike ibetitsmike deleted the mike/fix-mcp-screenshot-token-count branch January 12, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant