Skip to content

Flaky: IdempotencyRecordPurgeSweepTests lock-block probe times out on CI (sweep DELETE never blocked) #470

Closed

Description

@mforce

Evidence

Build-and-test run 31289913153 on PR #460 (commit 6ff9cee) failed:

Failed Cluckwork.Api.IntegrationTests.IdempotencyRecordPurgeSweepTests.Sweep_DoesNotDeleteAnExpiredClaimStolenWhileTheDeleteIsBlocked [10 s]
System.InvalidOperationException : sweep DELETE never blocked on the idempotency row lock within the timeout
  at WaitUntilSweepBlockedAsync (IdempotencyRecordPurgeSweepTests.cs:173)

The failure is in the test's own SETUP probe: it holds a row lock, starts the sweep, and polls pg_locks (or equivalent) until the sweep's DELETE is observed blocking — that observation never arrived within 10s on the CI runner. First occurrence in the last 15 runs on this branch; passes locally and on every other CI run.

Analysis

This is the arrange-phase of a #269-style race pin, not the guarantee itself — the sweep may simply not have reached its DELETE within the window on a loaded runner (Testcontainers Postgres + parallel suites). Shape matches the condition-based-waiting guidance: the poll is condition-based but the 10s ceiling is arbitrary relative to runner load.

Not a PR-#460 regression

The failing branch touches schema-docs guards, the docs generator, and the sim harness only — no idempotency, sweep, or persistence code.

Next step

Either raise/scale the block-observation timeout (cheap), or make the probe deterministic (e.g. sweep-side barrier/injection point) per docs/decisions/407-writing-a-guard.md. Related in spirit to #468 (refresh-race flake) but a different subsystem and a different failure mode (arrange-phase timeout vs guarantee-phase classification).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingepic-1.1

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions