Skip to content

fix(metrics): don't count a client-disconnect cancellation as a failed recall/reflect operation - #2185

Merged
nicoloboschi merged 1 commit into
vectorize-io:mainfrom
r266-tech:fix-metrics-cancellation-not-failure-2122
Jun 15, 2026
Merged

nicoloboschi merged 1 commit into
vectorize-io:mainfrom
r266-tech:fix-metrics-cancellation-not-failure-2122

Conversation

@r266-tech

Copy link
Copy Markdown
Contributor

The client-disconnect cancellation added in #2127/#2131 (issue #2122) left a metrics-accuracy gap.

In api_recall/api_reflect the engine call runs inside with metrics.record_operation(...), and run_cancellable_on_disconnect converts the engine's OperationCancelledError into HTTPException(499). That 499 propagates out through record_operation's blanket except Exception: success = False, so every abandoned/cancelled recall and reflect is counted as a failed operation on the documented hindsight.operation.total success label — inflating the recall/reflect failure rate (and any alerts built on it) the moment a client disconnects. The engine already treats cancellation as "not a failure to retry or report" (OperationCancelledError); record_operation was the one place still violating that.

Fix: recognise a client cancellation in record_operation and exclude it from hindsight.operation.total entirely — it's neither a success nor a failure, so it inflates neither rate, and its partial duration isn't recorded. Cancellation is detected via the exception's __cause__ chain (the HTTP layer raises HTTPException(499) from OperationCancelledError), not a bare status code, so an unrelated 499 is still recorded as a failure.

I kept the binary success label unchanged, so there's no docs/generated-file change. If you'd prefer explicit observability I'm happy to switch to a distinct cancelled outcome instead (that would also update monitoring.md + regenerate the skills docs mirror) — let me know which you'd rather have.

Regression tests cover: a direct OperationCancelledError and the HTTPException(499) from cancel path are excluded from the counter/histogram, while a bare 499 (no cancellation cause) is still recorded as a failure.

…d operation

run_cancellable_on_disconnect (added in vectorize-io#2127/vectorize-io#2131 for vectorize-io#2122) converts the
engine's OperationCancelledError into HTTPException(499), which propagates out
through record_operation's blanket `except Exception: success = False`, so every
abandoned recall/reflect was counted as a failure on hindsight.operation.total.

Exclude a client cancellation from the counter entirely (neither success nor
failure), detecting it via the exception's __cause__ chain so an unrelated 499
is still recorded as a failure. Adds regression tests.
@nicoloboschi
nicoloboschi merged commit c3970cf into vectorize-io:main Jun 15, 2026
77 of 78 checks passed
nicoloboschi added a commit that referenced this pull request Oct 30, 2026
…lled" (#6212)

* fix(metrics): count client-disconnect cancellations as success="cancelled" (#6126)

A recall/reflect the client abandoned was dropped from
hindsight.operation.total and .duration entirely (#2185), so there was no
way to see how often clients gave up or how long they waited first. It is
now recorded with success="cancelled": counted like any other operation,
with its time-until-cancel in the duration histogram, while
success="false" stays a server-failure-only signal.

* chore(docs-skill): regenerate monitoring reference
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants