Skip to content

fix(llm): retry a 200 that ends with finish_reason=error instead of reading it as no tool call (#5612) - #5751

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/5612-finish-reason-error
Oct 19, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/5612-finish-reason-error

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #5612.

Problem

OpenRouter reports an upstream that died mid-generation as HTTP 200 with finish_reason: "error", no tool calls, and the cause on the choice. The tool-call path read choices[0] directly, so this became an empty tool-call result: reflect treated a transport failure as "the model chose no tool", spent its single loop-wide retry, then answered without the forced recall / search_observations step. Nothing recorded which upstream failed.

Fix

  • openai-compatible: a finish_reason="error" choice raises a retryable ProviderResponseError (with the choice's error and the served upstream), checked in the helper every path reads its first choice through. call_with_tools now uses that helper and retries ProviderResponseError with backoff inside its existing attempt budget, like call() already did. call() also stops accepting partial text from an errored generation.
  • litellm: LiteLLM maps "error" to "stop" but keeps native_finish_reason and error in the choice's provider_specific_fields; the same check reads them there, and both call paths retry it.
  • tracing: spans carry hindsight.provider.served_by (OpenRouter's response provider), and every retried attempt is recorded as a failed span + success=False metric with its billed usage — previously it left only a log line.

Behaviour changes

  • When every attempt errors, the tool call raises instead of returning an empty result (reflect's own capped LLM-error handling takes it from there).
  • The tool path now also rejects an error payload / empty choices with a clear ProviderResponseError (empty choices used to be an IndexError).

Tests

  • tests/test_openai_compatible_finish_reason_error.py: reporter-shaped responses — tool call retried then answered; raises naming the upstream when retries run out; call() rejects partial text; retried attempt traced with upstream + usage; OTel span attribute; LiteLLM (call and call_with_tools) through LiteLLM's real response conversion.
  • Tool-path fakes in 8 test files now set error=None (the path never checked it before).

Not in scope (from the issue's ask 3): surfacing "a forced step was skipped" in reflect / refresh result metadata, and a per-step forced retry.

…eading it as no tool call (#5612)

OpenRouter reports an upstream that died mid-generation as HTTP 200 with
finish_reason="error". The tool path read it as an empty tool-call result, so
reflect skipped forced retrieval steps instead of retrying the request.

- openai-compatible: a finish_reason=error choice raises a retryable
  ProviderResponseError, checked where every path reads its first choice; the
  tool path now goes through that check and retries it with backoff.
- litellm: it rewrites finish_reason "error" to "stop"; the original is read
  from provider_specific_fields, and both call paths retry it.
- tracing: spans carry the served upstream (hindsight.provider.served_by), and
  each retried attempt is recorded as a failed span + metric with its usage.
- tests: fake responses on the tool path set error=None (the path never
  checked before); the empty-choices tool test expects the clear error.
@nicoloboschi
nicoloboschi merged commit 7127ae1 into main Oct 19, 2026
117 checks passed
@nicoloboschi
nicoloboschi deleted the fix/5612-finish-reason-error branch October 19, 2026 10:30
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant