Skip to content

fix: flaky tests #5804, #5806, #5830; name the cause of a failed bank delete (#5805) - #5838

Merged
nicoloboschi merged 2 commits into
mainfrom
fix/flaky-tests-5804-5805-5806-5830
Oct 20, 2026
Merged

nicoloboschi merged 2 commits into
mainfrom
fix/flaky-tests-5804-5805-5806-5830

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #5804, #5806, #5830. Refs #5805 (made diagnosable, not fixed).

#5830: creating a mental model on a new bank returned a 500 on a deadlock (real bug)

The CI deadlock detail: one process waits for ShareLock on memory_units, the other for ShareLock on a virtual transaction. Creating a bank builds its vector indexes with a plain CREATE INDEX inside the caller's transaction. Another bank's CREATE/DROP INDEX CONCURRENTLY waits on that transaction, and the transaction waits on it back. create_bank_if_missing already retries this. The three writes that create the bank on their own connection did not: create_mental_model, create_knowledge_folder and create_knowledge_page. All three now run their transaction under retry_with_backoff. It rolled back, so a redo is safe. Indexing the page still happens after the commit.

Regression test test_write_that_creates_its_bank_retries_transient_deadlock (one case per path) injects one DeadlockDetectedError. All 3 cases fail without the fix and pass with it.

#5804: event-loop test failed on one machine stall

It asserted on the single longest gap, so one GC pause or CPU-steal stall on CI could fail it. It now counts the ticker's turns during the batch (>= 3). Checked: with the resolver's asyncio.sleep(0) removed, the test fails with 0 turns.

#5806: re-ingest + delete of linked documents deadlock

Neither operation runs index DDL. But planning any memory_units statement takes AccessShare on every bank's vector index. That means another xdist worker's CONCURRENTLY drop can deadlock with either side (the same pattern as #5752). The test now reruns a round, up to 3 times, only when the deadlock detail names a virtual transaction. A real row-lock cycle between the two operations still fails, and the failure now prints the deadlock detail. The CI log had none, so I can't prove which kind it was.

#5805: not fixed, made diagnosable

From the CI log: DELETE FROM documents in delete_bank hit the 60s command timeout (TimeoutError, empty message). During that same minute almost every other xdist worker stalled too. So it looks like a DB-wide lock queue, not something in this test. A likely chain is bank-create's plain CREATE INDEX ShareLock queued behind a CONCURRENTLY index build, with every writer queued behind it. Not proven. The error now carries the type (TimeoutError: ...) and the cause, so the next failure tells us more. The lasting fix is likely to stop building a new bank's indexes with a plain CREATE INDEX inside a transaction. That is bigger and left for a follow-up.

Also closed

… delete (#5805)

- Writes that lazily create their bank inside their own transaction (create
  mental model, knowledge folder, knowledge page) now retry that transaction
  when it loses a deadlock. The bank create builds the bank's vector indexes
  with a plain CREATE INDEX, whose ShareLock on memory_units can deadlock with
  another bank's CONCURRENTLY index build/drop; the API returned a 500 (#5830).
- delete_bank's error now names the exception type and chains it. A lock-wait
  timeout has an empty str(), so the message said nothing (#5805).
- Event-loop test counts ticker turns during the batch instead of timing the
  longest gap, which one CI machine stall could fail (#5804).
- Linked-documents race test reruns a round only when the deadlock names a
  virtual transaction (another worker's index DDL, a third party), and prints
  the deadlock detail on any other failure (#5806).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_entity_resolver_candidate_cap::test_scoring_loop_keeps_the_event_loop_responsive on a loaded CI runner

1 participant