Move the step-up grant registry to a shared store before running >1 replica (follow-up to #308) #338

Coding brief — read first. Start with epic #530 (the canonical decision record); this is slice S2 of that epic and depends on #543 (the shared-state ports). This body is complete and current as of 2026-08-16, folding every review correction — no comments to cross-reference. Repo conventions in AGENTS.md apply (Version++, the #407 migration freeze, #417 schema-doc regeneration, the guard-writing rules). Definition of done = the acceptance criteria at the end.

Split out of #308 while implementing it (PR #336). Slice S2 of epic #530.

Problem

#308's step-up grant provides two guarantees a bare stateless JWT cannot: single-use (replay protection) and logout revocation. Both are backed by InMemoryStepUpGrantRegistry — in-process state, one instance per process, chosen so #308 needed no new infrastructure.

Correct for a single serving instance; broken otherwise:

• More than one replica. A grant consumed on replica A is still unused on B, so a single-use grant becomes reusable once per replica; a logout recorded on A does not revoke grants on B.
• Process restart. Both tables drop, which widens what an unexpired grant can do — it fails open, like the per-IP limiter's buckets (Security: rate limiting on anonymous auth endpoints (login/refresh) #143).

Blast radius is bounded by the 5-minute grant lifetime and the access token is still required alongside it, so this is defense-in-depth degradation, not an open door — but it is one of the four blockers to running a second replica.

This issue splits in two — the two tables have opposite storage shapes

1. Grant replay (_consumed) → the shared store

One entry per privileged admin operation, 5-minute TTL, self-expiring. Cache-shaped. It goes behind the claim-once port from #543 (Redis, in-process fallback). Losing it only lets an already-used grant be replayed within its remaining lifetime — what already happens on a restart today. When Redis is unreachable it fails closed (privileged operations refused when single-use cannot be proven).

2. Logout epoch (_loggedOutAt) → a user-row column, NOT the shared store

InMemoryStepUpGrantRegistry deliberately never prunes this table — one entry per user who has ever logged out, bounded by user count, never expiring. It is not a cache; it is a durable per-user revocation marker, the same shape as CredentialEpoch (#364), already read fresh from the DB on every authenticated request. In Redis it would face silent eviction under maxmemory pressure — load-dependent, invisible, on exactly the never-expiring key an LRU policy discards first. A restart at least shows in a deploy log.

So it becomes a column beside CredentialEpoch. This is the migration this slice owns (#543 is schema-free): add the column, backfill, regenerate docs/schema/ in the same PR (#417). One migration per change (#407). Redis being unreachable does not affect it.

The atomicity consequence — must not be discovered during implementation

TryConsumeIfNotLoggedOut currently checks the epoch and consumes the grant id under one lock, and this repo has already shipped the bug that comes from splitting them (PR #336, third round — the race is traced in IStepUpGrantRegistry's header):

T1 (validate): logout epoch not yet set -> ok
T2 (logout):   records the logout
T1 (validate): consumes the grant       -> admitted

With the epoch in Postgres and the grant id in Redis, no single lock spans both stores. Resolution is ordering:

Consume in Redis first, then read the epoch, and refuse on mismatch.

A logout landing between the two is caught by the read; one landing after it is genuinely after admission — the guarantee the lock gives today. Checking the epoch before consuming leaves the original race intact, so the order is load-bearing and must be commented as such. The cost: a logout-revoked grant burns its replay slot, which the interface currently avoids while noting it is "harmless today (it is refused either way)" — that note is now the justification. Keep StepUpAuthTests' spy-registry test; it exists to catch a re-split.

Non-enumerating contract stays: "revoked by logout" and "replayed" must remain indistinguishable to the caller, so the two-store split must not leak which one refused — no distinct error, latency, or log line.

Acceptance criteria

• A grant consumed on one replica is rejected as a replay on every other replica.
• A logout on one replica revokes outstanding grants for that user on every other replica.
• A process restart does not widen what an unexpired grant can do (replay fails closed on Redis loss; the logout epoch is durable in the DB).
• The claim-once store is pruned/TTL-bounded and cannot grow without bound; the logout-epoch column is bounded by user count.
• Consume-before-check ordering is implemented and commented; the spy-registry test still catches a re-split.
• Single-instance behaviour and Require recent step-up authentication for privileged user administration #308's existing tests stay green.
• AGENTS.md records the multi-replica prerequisites in one place (handled by Docs: multi-farm tenancy ADR + AGENTS.md / GLOSSARY sync #537).

Verify

dotnet test Cluckwork.sln
tools/schema-docs/generate.sh --check