Skip to content

fix(init): fail fast when model init blocks instead of hanging forever - #2014

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/model-init-timeout-1897
Jun 5, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/model-init-timeout-1897

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #1897

Problem

Model/connection initialization had no wall-clock cap. If embeddings, the cross-encoder, or LLM verification blocked — e.g. an offline HuggingFace download or an unreachable provider — asyncio.gather in MemoryEngine.initialize() (memory_engine.py:2313) never returned. The daemon got stuck in a third state: not started, not errored, just hung. From the caller's side the memory operation hangs forever with no way to recover.

The lazy reranker path CrossEncoderReranker.ensure_initialized() (reranking.py:156) had the same gap, surfacing the hang on the first recall instead of at startup.

Fix — fail fast, no silent fallback

  • New static config HINDSIGHT_API_MODEL_INIT_TIMEOUT (default 300s — generous enough for legitimate first-time model downloads on slow connections).
  • Wrap the startup gather and the lazy reranker init with asyncio.wait_for. On timeout, raise a RuntimeError that names the likely cause and points at the env var.
  • No graceful degradation / RRF fallback — consistent with the prior fix(api-slim): gracefully handle model init failures #1879 discussion that silent degradation is the wrong approach.

Deliberately kept minimal-scope: did not add the diagnostic _embeddings_initialized / _reranker_initialized state fields the issue floated — the reranker already tracks _initialized and they aren't needed for the fix.

Note: env var follows the project's HINDSIGHT_API_* convention (the issue suggested HINDSIGHT_MODEL_INIT_TIMEOUT).

Tests

tests/test_model_init_timeout.py:

  • timeout fires on a hung init → RuntimeError
  • fast init completes and marks the reranker initialized
  • default config value is 300s

Lint, format, ty type-check, and existing reranker-timeout tests all pass.

Model/connection initialization had no wall-clock cap: if embeddings, the
cross-encoder, or LLM verification blocked (e.g. an offline HuggingFace
download or an unreachable provider), `asyncio.gather` in
`MemoryEngine.initialize()` never returned and the daemon hung in a third
state — neither started nor errored. The lazy reranker path
(`CrossEncoderReranker.ensure_initialized()`) had the same problem on the
first request.

Wrap both with `asyncio.wait_for` capped by a new static config
`HINDSIGHT_API_MODEL_INIT_TIMEOUT` (default 300s, generous enough for
first-time model downloads). On timeout, raise a clear RuntimeError that
names the likely cause and points at the env var — no silent fallback.

Fixes #1897
@nicoloboschi
nicoloboschi merged commit 4f50034 into main Jun 5, 2026
236 of 240 checks passed
@nicoloboschi
nicoloboschi deleted the fix/model-init-timeout-1897 branch June 5, 2026 14:18
cdbartholomew added a commit that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daemon hangs indefinitely when model init blocks (no timeout on asyncio.gather in initialize())

1 participant