Skip to content

fix(operations): let provider switches survive pasted images in both directions - #31

Merged
Juliusolsson05 merged 4 commits into
mainfrom
fix/provider-switch-image-attachments
Sep 18, 2026
Merged

Juliusolsson05 merged 4 commits into
mainfrom
fix/provider-switch-image-attachments

Conversation

@Juliusolsson05

@Juliusolsson05 Juliusolsson05 commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

One user episode on 2026-09-18 hit two defects in this package in sequence, and the debug bundle records both:

  1. OpenCode → Claude (bug(claude): native-resume projector emits foreign image and document parts verbatim instead of Claude image/document blocks #29). The Claude native-resume projector copied the neutral image value verbatim, so an OpenCode {type:'file', mime, url:'data:…', id:'prt_…', …} part landed in the Claude transcript unchanged. Claude answered "I can't view images in this session" twice, and on the next prompt the API rejected the whole conversation: 400 messages.985.content.1: Input tag 'file' found using 'type' does not match any of the expected tags: …, 'document', 'image', …. The block stays in history, so the session was unusable from then on. Codex input_image items take the same path.
  2. Claude → OpenCode (bug(operations): shrink ladder cannot remove message attachments and never revisits protected turns, so one pasted image makes a conversation unfittable #28). Switching back failed with ConversationUnfittableError: … the smallest complete suffix the ladder can produce is 549785 characters against the 288,000-character budget. The transcript (1,100 entries, 2,458,176 estimated characters) ended with a 15-character prompt carrying a 549,526-character base64 screenshot, two turns after a prompt carrying the 713,997-character opaque file part from defect 1. The ladder cleared 256 outputs, trimmed 3 inputs, dropped 15 turns — and had no rung that reads message content, while the drop rung's fallback kept the protected newest turn whole.

Implemented behaviour

operations/shrink.ts (commit 1, fixes #28)

  • Rung 3, clearAttachments: replaces image, document and opaque content items inside messages with [<image|document|attachment> omitted during provider switch], oldest first, under the same recent-turn protection and net-savings guard as rung 2. Text items are never touched. The rung sits between clearing outputs and trimming inputs because a stale screenshot is consumed input like a stale tool output, while tool-call inputs hold the session's edits. Report: clearedAttachments (per item), clearedAttachmentChars (net).
  • Two passes. If the drop rung cannot fit any complete turn, the protection is lifted (keepRecentTurns: 0) on the pre-drop conversation and the drop rung runs again; the ladder throws only if the newest turn's own text still exceeds the budget. Re-running on the pre-drop conversation matters: once recent payload is on the table, old turns the first attempt discarded may fit. The lift does not simply run all three clearing rungs: each stops only when the whole conversation fits, which inside the lifted range it rarely can, so that cleared every recent tool output on its way to the image that was the cause. liftProtection measures the full lift, then tries the cheaper rung subsets in ladder-cost order and keeps the first that fits and drops no more entries. Report: liftedRecentTurnProtection, true only when the second pass removed something, read from the counts rather than object identity.
  • clearToolResults and trimToolInputs are now idempotent: the output placeholder (matched whole, not by prefix) and the trim marker are terminal. The second pass exposed that a second visit "saved" a character or two, counted one loss twice, rewrote the trim marker to quote the previous marker's length, and flipped the lift flag on a conversation where nothing real was removed.
  • Rungs renumbered in comments (inputs 4, drop 5). The character estimate is unchanged; bug(operations): character estimate charges a base64 image at its serialized length, about 100× its token cost #30 records why it charges an image at ~100× its token cost and what a fix needs.

claude/project/nativeResume.ts (commit 2, fixes #29)

  • claudeMessageContent converts foreign image/document values: a Claude-shaped value passes through; a base64 data URL in url (OpenCode) or image_url (Codex) becomes the observed {type:'image'|'document', source:{type:'base64', media_type, data}} block for the API's closed image set (png, jpeg, gif, webp) and application/pdf, when the payload is plain base64 and an image is within the API's 5 MB base64 limit, recorded as native-resume.content.<kind>.repaired; anything else is dropped with the existing dropped change, and a message left empty is dropped as before. Remote URLs are not guessed into source.type: 'url' because no observed Claude fixture has that shape. An opaque item carrying such a data URL is re-encoded too, so a Claude transcript the old projector already poisoned is healed by a Claude → Claude duplicate or rewind.

Design decisions

  • Provider-neutral ladder, still. Rung 3 treats every non-text content item as payload and never inspects a provider's value. An opaque item counts because every cross-provider projector drops it on arrival, so leaving it charged the budget for bytes the target never sees.
  • The protection is a preference, not a refusal. The design already relaxed it for single-turn conversations on the argument that declaring a session unfittable while nine tenths of it is stale payload is a worse answer than the evidence supports. The second pass is that argument applied whenever the protected suffix is what does not fit. It is reported, because hosts have relayed the protection to users as a promise.
  • No size in the attachment placeholder. A base64 length tells the model nothing; the numbers are in the report for the host and the user.
  • Evidence-gated Claude blocks. Only shapes present in claude-message-block-image / claude-message-block-document are emitted.

Tests

Written first and failing first (14 red, then green). Census caveat 1 (redacted fixtures prove shape, not size) means every size-driven case is built in the test from the recorded shape; pastedImageTailConversation() in testing/engine/fixtureConversations.ts documents the real transcript it scales down, and the shared entry builders replaced a private copy in contextBudget.test.ts.

  • shrink.test.ts: clearAttachments (oldest first, protection honoured, all three content kinds, text untouched, net-savings guard, no-op when fitting); recorded-shape regression for bug(operations): shrink ladder cannot remove message attachments and never revisits protected turns, so one pasted image makes a conversation unfittable #28 (fits, both attachments cleared, lift reported, drop rung still fires, newest prompt kept with placeholder, never a fragment); lift for a final turn dominated by tool output; no lift when dropping suffices (protected entries identical); still throws when the newest turn's own text exceeds the budget.
  • contextBudget.test.ts: planConversationContext(…, { allowSourceTurns: false }) returns shrunk with the extended report for the recorded shape.
  • nativeResumeProjection.test.ts: OpenCode file part → Claude image block (no "type":"file", no prt_ id on the wire); Codex data URL → image block, remote URL dropped with a change; Claude-shaped passthrough + PDF → document block; empty message dropped.

Verification

  • npm run check on Node 24: contract satisfied, typecheck clean, 26 files / 152 tests passed, packed surface verified. CI green on the final head.
  • The recorded transcript replayed through planConversationContext(…, 'opencode', 288_000, { allowSourceTurns: false }): shrunk in 70 ms, estimatedCharactersAfter: 32605, report { clearedResults: 256, clearedAttachments: 2, clearedAttachmentChars: 1263463, trimmedInputs: 3, droppedTurns: 13, droppedEntries: 1055, liftedRecentTurnProtection: true } (the six newest tool outputs survive; the first version of the lift cleared them and kept 6,975 characters); retained history opens on the marker then a complete user turn, and the newest user message is This [Image #1] + [image omitted during provider switch]. (Scratch script against a personal transcript, not committed.)
  • Host side: Juliusolsson05/agent-code PR (submodule bump, describeShrink, design doc) runs the app's provider-switch suite against this branch.

Review round

Four read-only reviewers across this PR and the host PR, no blocking findings. Fixed in ec514bc: the non-idempotent input trim and its false lift flag; the wasteful full lift (subset search); over-wide image media types and unvalidated payloads; healing of already-poisoned Claude transcripts; prefix-only placeholder detection; two rung references the renumbering missed; the header's stale design pointer; a census sentence credited with something it did not measure; missing tests (older turns survive the lift, PDF gate, mime fallback, direct rung idempotency). Every new regression test was mutation-checked: reverting its fix fails exactly that test.

Declined, with reasons: RFC 2397 parameters and upper-case ;BASE64, are dropped rather than parsed (safe direction, no recorded producer emits them). "Protection outranks the drop rung when the first pass fits" is recorded on #30, where the image overcharge that makes it common is addressed. The PR title keeps the operations scope although the second fix is claude; the repo merges with merge commits, so the title never becomes a commit subject.

Known limitations / follow-ups

Refs Juliusolsson05/agent-code#998, Juliusolsson05/agent-code#999

🤖 Generated with Claude Code

Juliusolsson05 and others added 3 commits September 18, 2026 13:55
Refs #28, #29, #30

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ction before refusing a shrink

A recorded Claude transcript (1,100 entries, 2,458,176 estimated characters)
could not be switched to a 288,000-character OpenCode target: after clearing
256 tool outputs, trimming 3 inputs and dropping 15 turns the ladder still
held a 549,620-character final turn — a 15-character prompt plus one base64
screenshot — and threw ConversationUnfittableError. No rung read message
content, and the drop rung's fallback kept the protected newest turn whole.

Rung 3 now replaces image, document and opaque content items inside messages
with a text placeholder, oldest first, under the same protection and
net-savings guard as the tool-output rung. The ladder runs its clearing rungs
a second time with the protection lifted when the drop rung cannot fit any
complete turn, on the pre-drop conversation so older turns that now fit are
kept, and throws only if the newest turn's own text is still too large. Both
are reported (clearedAttachments, clearedAttachmentChars,
liftedRecentTurnProtection) so no lossy step is silent. Clearing a tool
output is now idempotent, which the second pass requires for the report to
describe what the user lost rather than the rung's own arithmetic.

The character estimate is unchanged; #30 records why it overstates images.

Fixes #28
Refs Juliusolsson05/agent-code#998

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…f copying them verbatim

The Claude native-resume projector pushed every neutral image/document value
into the transcript unchanged. The OpenCode decoder wraps its native
{type:'file', url:'data:…'} part and the Codex decoder its
{type:'input_image', image_url} item, so an OpenCode → Claude switch wrote a
'file' block that the Claude API rejected on the very next prompt
("400 … Input tag 'file' found using 'type' does not match any of the
expected tags"), leaving the session unusable; before that the model could
not see the image at all.

Base64 data URLs in `url` or `image_url` become the observed Claude
{type:'image'|'document', source:{type:'base64', media_type, data}} block,
recorded as a repaired change. Claude-shaped values pass through as before.
Remote URLs and non-PDF documents are dropped with the existing change record
rather than guessed into an unobserved source shape, and a message left empty
by that is dropped as it always was.

Fixes #29
Refs Juliusolsson05/agent-code#999

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…mage re-encoding

Four read-only reviewers found no blocker and several real defects.

The second pass counted one loss twice. A tool input whose bulk is nested can
never get under the cap, so its string member was already the bare marker when
the lifted pass visited it again; re-truncating the marker saved one character,
rewrote "103 characters omitted" as the length of the previous marker, reported
three trims for two tool calls and set liftedRecentTurnProtection on a switch
where nothing real was removed. The trim marker is now terminal, as the output
placeholder already was, and that placeholder is matched whole rather than by
prefix so an output that merely begins with those words can still be cleared.
"The lift removed something" is read from the counts instead of from object
identity, which three separate return statements had been keeping true by
accident.

The lift also took more than it needed. Each rung stops only when the whole
conversation fits, which inside the lifted range it rarely can, so rung 2
cleared every recent tool output on its way to the pasted image that was the
cause: 256 to 262 outputs on the recorded transcript for no extra retained
history. liftProtection now measures the full lift, then tries the cheaper
rung subsets in ladder-cost order and keeps the first that fits and drops no
more entries. The recorded transcript keeps its six newest outputs and 32,605
characters instead of 6,975.

The Claude projector accepted any image/* and any payload. Images are now the
API's closed set (png, jpeg, gif, webp), the payload must be plain base64 and
within the 5 MB base64 limit, and everything else drops with a change record:
an HEIC or SVG part would otherwise have produced the same permanent 400 this
work exists to end. An opaque item carrying such a data URL is re-encoded too,
which lets a Claude transcript the old projector already poisoned be healed by
a Claude to Claude duplicate or rewind.

Also: two rung references the renumbering missed, the header's stale design
pointer, a census sentence credited with something it did not measure, and
tests for the pre-drop rerun, the PDF gate, the mime fallback and rung
idempotency. Every new regression test was mutation-checked. Plan marked
complete with the as-built notes and declined findings.

Refs #28, #29, #30, #32

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Juliusolsson05
Juliusolsson05 merged commit 7aae690 into main Sep 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant