[cdbartholomew]

Summary

Adds a pre-flight planner-estimate gate to retrieve_temporal_combined that
skips the temporal retrieval pass when the date filter would match an
excessive number of rows. This protects the database from queries that are
both expensive and statistically meaningless on banks with dense or
near-uniform date metadata.

Why

Phase 1 of retrieve_temporal_combined ranks rows by
COALESCE(occurred_start, mentioned_at, occurred_end) over the entire set
matching its WHERE clause, then keeps the top 50 per fact_type. This works
well when the filter is selective. It degrades nonlinearly when the
candidate set crosses ~100k rows for three reasons:

1. Sort spills from work_mem to disk (~10× slower per tuple under an
   external merge sort).
2. The top-50-per-fact_type sample stops being statistically meaningful —
   when hundreds of thousands of rows are ~tied on the sort key, the date
   ranking has no signal and the join back to memory_units produces
   near-random candidates that the downstream cross-encoder still has to
   rerank.
3. Phase 2's hash join probes the full table for the embedding distance,
   making the whole query I/O-bound.

A bank that accumulates dense, near-uniform date metadata — for example
because a retain pipeline stamps mentioned_at = NOW() on a large batch —
lands in this regime for any temporal recall that intersects the dense
zone. Profiling against a representative 660k-row bank with a 1-day window
showed:

• 33 s execution time
• 30 MB external sort to disk
• ~2.8M shared buffer touches per query
• ~184k disk reads per query

Under N-way concurrent recall this can saturate the primary.

How

Before issuing the real temporal query, ask the planner via EXPLAIN (FORMAT JSON) how many rows the Phase 1 filter would match. The planner
walks its statistics and returns an estimate in milliseconds with no row
I/O. Above the configurable threshold
(HINDSIGHT_API_TEMPORAL_RECALL_MAX_ESTIMATED_ROWS, default 50,000), the
function returns an empty per-fact_type dict and lets semantic+BM25 carry
the recall. The caller in retrieve_all_fact_types_parallel already
tolerates an empty temporal arm (it's one of several parallel retrieval
arms).

Threshold

filter match count	sort	per-query	top-50 signal
< 50k	in-memory	< 2 s	meaningful
50k – 100k	disk	5 – 15 s	weak
> 100k	disk	> 15 s	near-random

50k is the conservative lower bound — it doesn't false-positive on healthy
banks (planner estimates tend to over-shoot real counts on partial-index
filters, which is the safe direction here: the gate only over-blocks,
never under-blocks). Estimate accuracy verified at ~99.7% on the
catastrophic case (planner 644,518 vs actual 646,482) and 2–38× over-shoot
on healthy cases, all of which remain well under the threshold.

Failure modes

• EXPLAIN can't be parsed (driver quirk, permission error, dialect
  drift) → fail open and let the temporal query run. Better a slow query
  than a silently-empty recall.
• Non-postgres backend (Oracle) → gate opts out entirely. The
  EXPLAIN FORMAT JSON format used here is pg-specific and the Oracle
  planner surface would need its own implementation.

Tests

Five unit tests under
hindsight-api-slim/tests/test_temporal_recall_circuit_breaker.py:

• gate fires above threshold (skips the real query)
• gate allows below threshold (real query runs)
• strict-inequality boundary (a refactor switching > to >= would
  silently change behavior — pinned)
• EXPLAIN-failure fail-open path
• non-postgres backend opt-out

All 5 pass. Broader regression sweep (112 tests across search/ and
db_abstraction) clean. Lint hook clean.

Test plan

• CI green
• Roll out behind the default 50,000 threshold; instrument logs to
  observe how often the gate triggers in real workloads
• If false-positives appear, raise the env var without redeploying

[nicoloboschi]

superseded by #1983