Repository navigation
fix: flaky tests #5804, #5806, #5830; name the cause of a failed bank delete (#5805) - #5838
Merged
Merged
Conversation
… delete (#5805) - Writes that lazily create their bank inside their own transaction (create mental model, knowledge folder, knowledge page) now retry that transaction when it loses a deadlock. The bank create builds the bank's vector indexes with a plain CREATE INDEX, whose ShareLock on memory_units can deadlock with another bank's CONCURRENTLY index build/drop; the API returned a 500 (#5830). - delete_bank's error now names the exception type and chains it. A lock-wait timeout has an empty str(), so the message said nothing (#5805). - Event-loop test counts ticker turns during the batch instead of timing the longest gap, which one CI machine stall could fail (#5804). - Linked-documents race test reruns a round only when the deadlock names a virtual transaction (another worker's index DDL, a third party), and prints the deadlock detail on any other failure (#5806).
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5804, #5806, #5830. Refs #5805 (made diagnosable, not fixed).
#5830: creating a mental model on a new bank returned a 500 on a deadlock (real bug)
The CI deadlock detail: one process waits for ShareLock on
memory_units, the other for ShareLock on a virtual transaction. Creating a bank builds its vector indexes with a plainCREATE INDEXinside the caller's transaction. Another bank'sCREATE/DROP INDEX CONCURRENTLYwaits on that transaction, and the transaction waits on it back.create_bank_if_missingalready retries this. The three writes that create the bank on their own connection did not:create_mental_model,create_knowledge_folderandcreate_knowledge_page. All three now run their transaction underretry_with_backoff. It rolled back, so a redo is safe. Indexing the page still happens after the commit.Regression test
test_write_that_creates_its_bank_retries_transient_deadlock(one case per path) injects oneDeadlockDetectedError. All 3 cases fail without the fix and pass with it.#5804: event-loop test failed on one machine stall
It asserted on the single longest gap, so one GC pause or CPU-steal stall on CI could fail it. It now counts the ticker's turns during the batch (>= 3). Checked: with the resolver's
asyncio.sleep(0)removed, the test fails with 0 turns.#5806: re-ingest + delete of linked documents deadlock
Neither operation runs index DDL. But planning any
memory_unitsstatement takes AccessShare on every bank's vector index. That means another xdist worker's CONCURRENTLY drop can deadlock with either side (the same pattern as #5752). The test now reruns a round, up to 3 times, only when the deadlock detail names a virtual transaction. A real row-lock cycle between the two operations still fails, and the failure now prints the deadlock detail. The CI log had none, so I can't prove which kind it was.#5805: not fixed, made diagnosable
From the CI log:
DELETE FROM documentsindelete_bankhit the 60s command timeout (TimeoutError, empty message). During that same minute almost every other xdist worker stalled too. So it looks like a DB-wide lock queue, not something in this test. A likely chain is bank-create's plainCREATE INDEXShareLock queued behind a CONCURRENTLY index build, with every writer queued behind it. Not proven. The error now carries the type (TimeoutError: ...) and the cause, so the next failure tells us more. The lasting fix is likely to stop building a new bank's indexes with a plainCREATE INDEXinside a transaction. That is bigger and left for a follow-up.Also closed