Fix Responses streaming usage metadata

[jhaynie]

Summary

• emit gateway metadata for SSE streams that do not include a terminal [DONE] marker
• route Responses streams through the Responses extractor when api_type is stored as a string
• extract Responses API usage from top-level usage and response.incomplete terminal events

Validation

• go test ./...

Summary by CodeRabbit

• Bug Fixes

  • Gateway billing metadata now properly emitted for all streaming responses, including those without terminal markers.
  • Improved token usage extraction from incomplete streaming events and alternative response structures.
  • Enhanced API type detection for flexible response format handling.

• Tests

  • Added streaming validation and API type dispatch verification tests.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 44960394-7efc-4378-8e1c-1345c2e86142

📥 Commits

Reviewing files that changed from the base of the PR and between b016666 and ff54084.

📒 Files selected for processing (7)

• autorouter.go
• autorouter_test.go
• providers/openai_compatible/multiapi.go
• providers/openai_compatible/responses_streaming_extractor_test.go
• providers/openai_compatible/responses_test.go
• streaming.go
• streaming_test.go

📜 Recent review details

🔇 Additional comments (7)

autorouter.go (1)

416-420: Good fix for non-terminal SSE streams.

This broadens gateway.metadata emission to every SSE response instead of only streams that buffered data: [DONE], while still keeping metadata ahead of any held terminal event.

autorouter_test.go (1)

1497-1573: Nice regression coverage for the no-[DONE] case.

This test exercises the exact SSE shape that previously missed gateway.metadata and checks the emitted payload at the HTTP boundary.

streaming.go (1)

205-210: Good expansion of Responses streaming usage handling

The updated event shape and extraction path correctly support both response.completed and response.incomplete, including top-level usage fallback.

Also applies to: 341-374

streaming_test.go (1)

715-738: Nice coverage additions for new Responses usage paths

These new cases cover both top-level usage and response.incomplete, matching the intended extraction behavior.

providers/openai_compatible/responses_test.go (1)

1761-1800: String-based api_type dispatch test is solid

This test closes the gap for context values stored as plain strings and verifies end-to-end routing and usage extraction.

providers/openai_compatible/responses_streaming_extractor_test.go (1)

88-124: Great targeted tests for non-[DONE] and incomplete terminal usage

These additions directly validate the new streaming metadata behavior for both event shapes.

providers/openai_compatible/multiapi.go (1)

104-105: ⚡ Quick win

The panic concern is unfounded—APIType is a string-derived type and is comparable.

APIType is defined as type APIType string in apitype.go, which means it's a comparable type in Go. Direct equality comparison on == cannot panic; the code at lines 104-105 is safe as written. All values stored in Custom["api_type"] are either APIType or string, both of which are comparable.

The suggested type switch refactor, while more explicit, is unnecessary for safety. The current comparison logic is correct.

			> Likely an incorrect or invalid review comment.

---

📝 Walkthrough

Walkthrough

This PR enhances OpenAI Responses API streaming support by expanding the ResponsesStreamEvent data model, updating usage extraction logic to handle both completed and incomplete events with flexible usage sources, relaxing API type dispatch to accept string values alongside typed enums, and changing gateway metadata event emission to trigger on any SSE response rather than only on terminal markers.

Changes

Responses API Streaming Support

Layer / File(s)	Summary
ResponsesStreamEvent Schema streaming.go	ResponsesStreamEvent struct expands to include id, object, model, status, and usage JSON-mapped fields beyond the original type field.
Flexible Usage Extraction streaming.go	ExtractUsageFromResponsesEvent now accepts both response.completed and response.incomplete events and extracts usage from either nested response payload or top-level event usage field.
Responses API Type Routing providers/openai_compatible/multiapi.go	StreamingMultiAPIExtractor routing accepts both llmproxy.APITypeResponses enum value and its string form "responses" when dispatching to responses streaming extraction.
Gateway Metadata Emission autorouter.go	Gateway metadata SSE event is now emitted when sseWriter is non-nil (SSE response detected) rather than only when sseWriter.HasTerminal() is true.
Usage Extraction Tests streaming_test.go	Added test cases for response.completed with top-level usage and response.incomplete with nested response usage.
Extractor and Routing Tests providers/openai_compatible/responses_streaming_extractor_test.go, providers/openai_compatible/responses_test.go	Tests cover top-level and incomplete usage extraction from ResponsesStreamingExtractor, and string-based api_type dispatch in MultiAPIExtractor.
Gateway Metadata Emission Tests autorouter_test.go	Test verifies gateway.metadata event is written even when upstream streaming lacks a terminal marker.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}