[nicoloboschi]

Summary

Cancel abandoned HTTP recall and reflect work via a cooperative cancellation token carried on RequestContext, instead of relying on asyncio task cancellation. Alternative to #2125 with an architecture that stops the most expensive stage and generalizes across endpoints.

Fixes #2122.

Problem

A /memories/recall (or /reflect) that outlives its client kept running server-side — ~2 CPUs for 60–95 s per abandoned recall on an 11.6K-unit bank — and abandoned requests accumulated toward RECALL_MAX_CONCURRENT until the instance starved (RSS 7.2 GB, CPU pinned at the container cap for hours, from a single retry-looping client). See #2122.

Why a token, not asyncio task cancellation

The recall pipeline's heaviest stage — cross-encoder reranking (flashrank) — runs in a ThreadPoolExecutor via run_in_executor. Cancelling the awaiting asyncio task only unblocks the await; the worker thread keeps burning CPU to completion. So task cancellation (the approach in #2125) returns 499 and frees the recall semaphore while an uncancellable thread still pins a CPU — RECALL_MAX_CONCURRENT stops bounding actual CPU.

Instead, a CancellationToken is threaded through RequestContext (already passed into every engine operation) and checked at stage / iteration boundaries, bailing out before dispatching the next expensive stage.

Changes

• New hindsight_api/cancellation.py: CancellationToken (one-shot, awaitable, idempotent) + OperationCancelledError.
• RequestContext gains a cancellation field and a raise_if_cancelled() helper (no-op when no token attached).
• Recall checkpoints in recall_async / _search_with_retries: pre-retrieval, pre-rerank, and pre-enrichment (chunk/entity/source-fact fetches).
• Reflect checkpoints: an early check in reflect_async, plus a cancel_check hook in run_reflect_agent invoked between agent iterations (and the nested recall tool already checks at its own boundaries since it shares the RequestContext).
• Shared HTTP wiring: run_cancellable_on_disconnect() attaches the disconnect-driven token (1 s poll, restores the prior token on exit) and maps OperationCancelledError → 499. Both the recall and reflect handlers call it — no duplicated try/except.
• Scoped to HTTP recall + reflect: internal callers (consolidation, MCP, mental-model triggers) pass no token, so every checkpoint is a no-op and behavior is unchanged.

Honest limitation

Cooperative: it cannot interrupt a computation already inside a worker thread. If the client disconnects mid-rerank, we finish that one rerank then abort at the next checkpoint. The win is that an abandoned request never starts rerank (if the client left during retrieval), never runs post-rerank enrichment, and a reflect loop stops between iterations instead of spending every remaining LLM round-trip — and RECALL_MAX_CONCURRENT keeps bounding real work. Truly bounding mid-rerank would need chunked token checks inside the predict loop or a killable subprocess (out of scope). A deadline-based driver (for the consolidation-contention half of #2122) is a trivial extension since the token lives on RequestContext.

Relationship to #2125

Alternative implementation of the same fix. #2125 wraps recall in a disconnect watcher and cancels the asyncio task; this PR uses a checkpoint token. The token approach stops the threadpool rerank stage from starting (which task cancellation can't reach), avoids early-releasing the concurrency slot while a thread runs on, reuses a carrier every engine operation already receives, and covers reflect with the same mechanism. Pick one.

Tests

uv run pytest tests/test_recall_cancellation.py -q   # 13 tests, all green
uv run ruff check hindsight_api/ && uv run ty check hindsight_api/api/http.py

Covers the token primitive, RequestContext integration, the disconnect driver (attach/restore/abort), the shared run_cancellable_on_disconnect helper (result-on-connected, 499-on-disconnect), and an ASGI end-to-end (disconnect → aborted staged work → 499). Existing reflect suites (test_reflect_agent, test_per_operation_llm_config, etc.) remain green — cancel_check defaults to None.

[Sanderhoff-alt]

I agree this is the better implementation. The cooperative cancellation token fits the recall pipeline better than my outer asyncio-task cancellation approach, especially around threadpool reranking.

I'll close my PR in favor of this one.