Repository navigation
fix(metrics): don't count a client-disconnect cancellation as a failed recall/reflect operation - #2185
Merged
nicoloboschi merged 1 commit intoJun 15, 2026
Conversation
…d operation run_cancellable_on_disconnect (added in vectorize-io#2127/vectorize-io#2131 for vectorize-io#2122) converts the engine's OperationCancelledError into HTTPException(499), which propagates out through record_operation's blanket `except Exception: success = False`, so every abandoned recall/reflect was counted as a failure on hindsight.operation.total. Exclude a client cancellation from the counter entirely (neither success nor failure), detecting it via the exception's __cause__ chain so an unrelated 499 is still recorded as a failure. Adds regression tests.
nicoloboschi
approved these changes
Jun 15, 2026
cdbartholomew
added a commit
that referenced
this pull request
Oct 20, 2026
nicoloboschi
added a commit
that referenced
this pull request
Oct 30, 2026
…lled" (#6212) * fix(metrics): count client-disconnect cancellations as success="cancelled" (#6126) A recall/reflect the client abandoned was dropped from hindsight.operation.total and .duration entirely (#2185), so there was no way to see how often clients gave up or how long they waited first. It is now recorded with success="cancelled": counted like any other operation, with its time-until-cancel in the duration histogram, while success="false" stays a server-failure-only signal. * chore(docs-skill): regenerate monitoring reference
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The client-disconnect cancellation added in #2127/#2131 (issue #2122) left a metrics-accuracy gap.
In
api_recall/api_reflectthe engine call runs insidewith metrics.record_operation(...), andrun_cancellable_on_disconnectconverts the engine'sOperationCancelledErrorintoHTTPException(499). That 499 propagates out throughrecord_operation's blanketexcept Exception: success = False, so every abandoned/cancelled recall and reflect is counted as a failed operation on the documentedhindsight.operation.totalsuccesslabel — inflating the recall/reflect failure rate (and any alerts built on it) the moment a client disconnects. The engine already treats cancellation as "not a failure to retry or report" (OperationCancelledError);record_operationwas the one place still violating that.Fix: recognise a client cancellation in
record_operationand exclude it fromhindsight.operation.totalentirely — it's neither a success nor a failure, so it inflates neither rate, and its partial duration isn't recorded. Cancellation is detected via the exception's__cause__chain (the HTTP layer raisesHTTPException(499) from OperationCancelledError), not a bare status code, so an unrelated 499 is still recorded as a failure.I kept the binary
successlabel unchanged, so there's no docs/generated-file change. If you'd prefer explicit observability I'm happy to switch to a distinctcancelledoutcome instead (that would also updatemonitoring.md+ regenerate the skills docs mirror) — let me know which you'd rather have.Regression tests cover: a direct
OperationCancelledErrorand theHTTPException(499) from cancelpath are excluded from the counter/histogram, while a bare 499 (no cancellation cause) is still recorded as a failure.