feat: shared-state ports with Redis + in-process fallback (#543)

AGENTS.md

@@ -127,7 +127,7 @@ Reviewers: treat a hardcoded provider name in code, config, or a committed doc l
 
 **Run one serving instance.** Every instance runs `DurableJobWorker`, and its poll claims nothing — no `FOR UPDATE SKIP LOCKED`, no lease, no advisory lock. Four independent blockers must close before scaling, and they are not all in #271: background work has no single-runner guarantee (**#271**); `IStepUpGrantRegistry` is process-local, so a single-use step-up grant becomes usable once *per replica* (**#338** — the blocker with teeth); the IP-keyed auth limiters (#143) are in-process; and the per-account report concurrency cap (#311) is in-process. **#307 is closed and does not license scaling.**
 
-**Do not extend that list from memory** — it was twice derived wrongly, both times a process-local limiter. Re-derive it by walking every `AddSingleton`/`AddHostedService` under `src/` plus every in-memory state primitive, then excluding deliberately. The run-then-exit verbs are unaffected: they never start hosted services. → [`271-single-serving-instance.md`](docs/decisions/271-single-serving-instance.md)
+**Do not extend that list from memory** — it was twice derived wrongly, both times a process-local limiter. Re-derive it by walking every `AddSingleton`/`AddHostedService` under `src/` plus every in-memory state primitive, then excluding deliberately. The run-then-exit verbs are unaffected: they never start hosted services. The four #543 shared-state registrations (`IConnectionMultiplexer` + the `IClaimOnceStore`/`IFixedWindowCounter`/`ILease` ports) are **not** new blockers — they are the shared store that will *close* #544/#545 and the #338 grant-replay once a caller is wired; Redis-backed they are multi-replica-safe, their in-process fallbacks a deliberate alarmed degradation. → [`271-single-serving-instance.md`](docs/decisions/271-single-serving-instance.md)
 
 ## Writing a guard (a test that asserts an invariant)
 

deploy/README.md

@@ -21,8 +21,11 @@ before that revocation can resurrect an epoch-0 child written during a mixed
 fleet window. Treat rollback as one forced re-login, not as a plain image
 downgrade.
 
-- `docker-compose.yml` — the production stack (app + Postgres), fronted by
-  Traefik for TLS. See the root [README](../README.md) to run it, and
+- `docker-compose.yml` — the production stack (app + Postgres + Redis), fronted
+  by Traefik for TLS. Redis backs the #543 shared-state ports; a real
+  multi-instance deploy points `SharedState__Redis__ConnectionString` at a
+  managed/shared Redis instead of the co-located sidecar. See the root
+  [README](../README.md) to run it, and
   [backup & restore](../docs/runbooks/backup-and-restore.md) for the dump/restore
   procedure.
 - `docker-compose.dev.yml` — just Postgres, for running the API from the IDE.

deploy/docker-compose.dev.yml

@@ -1,9 +1,13 @@
-# Dev dependencies only — Postgres, loopback-published on :5432 — so the API can
-# be run/debugged from the IDE (VS Code F5) against a real database.
+# Dev dependencies only — Postgres and Redis, loopback-published (:5432, :6379)
+# — so the API can be run/debugged from the IDE (VS Code F5) against real
+# backing services.
 # Usage: docker compose -f deploy/docker-compose.dev.yml up -d
 #
 # Credentials here are LOCAL DEV defaults, not secrets. The API reads its own
 # connection string from user-secrets (see .vscode/README or project setup).
+# Redis is OPTIONAL for local debug: leave SharedState:Redis:ConnectionString
+# unset in user-secrets to run on the in-process fallback, or set it to
+# "localhost:6379" to exercise the real Redis-backed shared-state path (#543).
 name: cluckwork-dev
 
 services:
@@ -23,5 +27,17 @@ services:
       timeout: 5s
       retries: 5
 
+  # #543 shared-state backend, loopback-published so an IDE-run API can reach it
+  # at localhost:6379. Same digest-pinned image as the deploy/sim/test stacks.
+  redis:
+    image: redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2
+    ports:
+      - "127.0.0.1:6379:6379"
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
 volumes:
   cluckwork-dev-pg18:

deploy/docker-compose.yml

@@ -48,6 +48,16 @@ services:
       # container healthcheck (Host: localhost) keeps working. HSTS + CSP + the
       # other security headers are emitted automatically outside Development.
       AllowedHosts: ${CLUCKWORK_HOST}
+      # #543: shared-state backend (grant-replay claim-once, auth limiters, report
+      # cap). Points at the co-located `redis` service so this reference stack
+      # boots the REAL Redis-backed path, matching a scaled deploy's config shape
+      # — even though this stack runs a single instance (where Redis vs in-process
+      # is functionally equivalent). Blank would be legal (the app degrades to the
+      # in-process fallback, Option B); this stack wires it on purpose. A real
+      # multi-instance deploy points this at its own shared Redis. The app forces
+      # AbortOnConnectFail=false, so an unreachable Redis degrades at runtime via
+      # the resilient decorators rather than failing the boot.
+      SharedState__Redis__ConnectionString: redis:6379
     # #266 — readiness probe. The image already carries this HEALTHCHECK; declared
     # here too so the stack's health is explicit and tunable. Runs the same
     # `healthcheck` verb inside the container (no curl/wget on the hardened image,
@@ -62,6 +72,11 @@ services:
     depends_on:
       db:
         condition: service_healthy
+      # #543 — the app boots the Redis-backed shared-state path; gate on the
+      # sidecar being healthy so the first request exercises Redis, not the
+      # in-process fallback.
+      redis:
+        condition: service_healthy
       # #263 — the schema is migrated by the one-shot `migrate` job below (run
       # under the owner credential), not at request time. app serves with
       # Database:MigrateOnStartup=false (Production appsettings default), so a
@@ -119,6 +134,21 @@ services:
       timeout: 5s
       retries: 5
 
+  # #543 — shared-state (Redis) backend for the scale-out ports (grant-replay
+  # claim-once, auth limiters, report-concurrency cap). Same digest-pinned image
+  # the Testcontainers RedisFixture and the #243 sim harness use, kept in
+  # lockstep. Co-located sidecar on the private compose bridge, no auth, not
+  # published — only `app` talks to it. A real multi-instance deploy replaces
+  # this with a managed/shared Redis (and TLS/auth on it); this reference stack,
+  # like its co-located plaintext Postgres, is single-host and never exposed.
+  redis:
+    image: redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
   traefik:
     # Prod-only TLS/host router. Local dev hits the API directly on :8080, so
     # traefik does not start with a bare `docker compose up`. Enable in prod

docs/decisions/271-single-serving-instance.md

@@ -115,6 +115,21 @@ This is the general lesson recorded in
 mean the **method** is wrong, so prefer "walk everything, exclude deliberately"
 over "list what I thought of".
 
+Also excluded, and specifically **not** new blockers: the four #543 shared-state
+registrations in `SharedStateRegistration` — `IConnectionMultiplexer` plus the
+three ports `IClaimOnceStore` / `IFixedWindowCounter` / `ILease`. These are the
+**shared store** that will *close* the in-process limiter blockers above, not
+extend them: the auth limiters (#544) move onto `IFixedWindowCounter`, the report
+cap (#545) onto `ILease`, and step-up grant replay (#338) onto `IClaimOnceStore`.
+Redis-backed, they hold their state in the one shared Redis, so they are
+multi-replica-safe by construction; `IConnectionMultiplexer` is a shared client.
+Their in-process fallbacks (`InProcess*`) are per-process state, but a
+**deliberate, alarmed** degradation (`SecurityEvents.SharedStateRedisUnavailable`;
+claim-once fails *closed*), reached only when Redis is blank or unreachable — not
+a silent single-instance trap. Caveat: #543 only lands the ports; #544/#545 and
+the #338 grant-replay move stay open until a caller is actually wired to them, at
+which point that wiring PR removes the corresponding blocker above.
+
 ## What does NOT license scaling
 
 **#307 (multi-replica HTTP write idempotency) is CLOSED**, so the request-path

src/Cluckwork.Api/Configuration/SharedStateOptions.cs

@@ -0,0 +1,17 @@
+namespace Cluckwork.Api.Configuration;
+
+// #543 — configuration for the shared-state (Redis) backend. A blank
+// ConnectionString means "no Redis": the app runs on the in-process
+// implementations (Option B, single instance). KeyNamespace prefixes every
+// Redis key so environments sharing one Redis do not collide.
+public sealed class SharedStateOptions
+{
+    public const string SectionName = "SharedState";
+    public RedisOptions Redis { get; init; } = new();
+
+    public sealed class RedisOptions
+    {
+        public string ConnectionString { get; init; } = "";
+        public string KeyNamespace { get; init; } = "cluckwork";
+    }
+}

src/Cluckwork.Api/Hosting/CluckworkSharedStateServiceCollectionExtensions.cs

@@ -0,0 +1,47 @@
+namespace Cluckwork.Api.Hosting;
+
+using Cluckwork.Api.Configuration;
+using Cluckwork.Infrastructure.SharedState;
+
+// #543 — binds SharedStateOptions, runs the serving-only connection-string
+// guard, then delegates registration to Infrastructure.
+internal static class CluckworkSharedStateServiceCollectionExtensions
+{
+    public static void AddCluckworkSharedState(
+        this IServiceCollection services, IConfiguration configuration, ProcessRole role)
+    {
+        var options = configuration.GetSection(SharedStateOptions.SectionName)
+            .Get<SharedStateOptions>() ?? new SharedStateOptions();
+
+        EnsureSharedStateConnectionValid(options.Redis.ConnectionString, role);
+
+        // The guard above owns the serving-fail, so the helper never needs to
+        // throw — a one-shot verb with a bad string degrades to in-process.
+        services.AddCluckworkSharedState(
+            options.Redis.ConnectionString,
+            options.Redis.KeyNamespace,
+            failOnMalformed: false);
+    }
+
+    // #543/#347 — SERVING-ONLY boot guard: a serving process with a set-but-
+    // malformed Redis connection string must fail loudly rather than silently
+    // run single-instance on the in-process fallback. Blank is fine (Option B,
+    // deliberate single-instance). A one-shot verb (migrate/seed/…) never uses
+    // shared state, so it is skipped — making this eager for every role would be
+    // a fresh #331. Discovered by ServingGuardCoverageTests (an `Ensure*` method
+    // in this namespace); its ProcessRoleGuardTests row proves it fires.
+    private static void EnsureSharedStateConnectionValid(string? connectionString, ProcessRole role)
+    {
+        if (role is not ProcessRole.Serving)
+            return;
+        if (string.IsNullOrWhiteSpace(connectionString))
+            return;
+        if (SharedStateRegistration.IsWellFormedConnectionString(connectionString))
+            return;
+
+        throw new InvalidOperationException(
+            $"{SharedStateRegistration.MalformedConnectionStringMessagePrefix} (no endpoint "
+            + "host:port was found). Fix the value, or leave it blank to run single-instance "
+            + "on the in-process implementations.");
+    }
+}

src/Cluckwork.Api/Program.cs

@@ -61,6 +61,8 @@
     builder.Configuration, processRole);
 builder.Services.AddCluckworkEdgeSecurity(rateLimiting.TrustedProxies);
 
+builder.Services.AddCluckworkSharedState(builder.Configuration, processRole);
+
 builder.Services.AddCluckworkFeatures(builder.Configuration);
 
 // #307 — lease duration / max-wait bounds for the idempotency claim protocol.

src/Cluckwork.Application/Common/SecurityEvents.cs

@@ -55,4 +55,11 @@ public static class SecurityEvents
     // policy (#217) — that budget guards log-pipeline volume, not a
     // credential, so its rejections carry no security-event line.
     public const string RateLimitRejected = "Auth.RateLimitRejected";
+
+    // Fires when a shared-state (Redis) operation throws and the caller
+    // degrades: grant-replay fails closed (denies), the auth limiter and the
+    // report-concurrency lease fall back to their in-process implementations.
+    // A deployment backend should alert on this — a limiter silently stuck in
+    // fallback is the failure this event exists to make visible (#543).
+    public const string SharedStateRedisUnavailable = "SharedState.RedisUnavailable";
 }

src/Cluckwork.Infrastructure/Cluckwork.Infrastructure.csproj

@@ -26,6 +26,7 @@
     <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="10.*" />
     <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="8.*" />
     <PackageReference Include="Serilog.Extensions.Logging" Version="10.*" />
+    <PackageReference Include="StackExchange.Redis" Version="2.*" />
   </ItemGroup>
 
   <ItemGroup>

src/Cluckwork.Infrastructure/SharedState/IClaimOnceStore.cs

@@ -0,0 +1,22 @@
+namespace Cluckwork.Infrastructure.SharedState;
+
+// #543 — single-use claim with TTL (port for grant replay, #338).
+//
+// A key can be claimed exactly once per TTL window. The first caller wins;
+// every other caller is refused while the claim is live, and the key becomes
+// claimable again once the TTL has elapsed. Time comes exclusively from the
+// injected <see cref="TimeProvider"/> — implementations must never read a
+// wall clock directly, so tests can drive expiry with a fake clock.
+internal interface IClaimOnceStore
+{
+    /// <summary>
+    /// Attempts to claim <paramref name="key"/> for the duration
+    /// <paramref name="ttl"/>.
+    /// </summary>
+    /// <returns>
+    /// <see langword="true"/> the first time the key is claimed and again once
+    /// the previous claim's TTL has elapsed; <see langword="false"/> while an
+    /// unexpired claim on the key exists.
+    /// </returns>
+    bool TryClaim(string key, TimeSpan ttl);
+}

src/Cluckwork.Infrastructure/SharedState/IFixedWindowCounter.cs

@@ -0,0 +1,23 @@
+namespace Cluckwork.Infrastructure.SharedState;
+
+// #543 — atomic increment within a fixed time window (port for the auth
+// limiters, #544, and the report concurrency cap, #545).
+//
+// The window for a key is the half-open interval
+// [floor(now / window), floor(now / window) + window). The count accumulates
+// for the duration of the current window and resets to zero when the window
+// rolls over. Time comes exclusively from the injected <see cref="TimeProvider"/>
+// — implementations must never read a wall clock directly, so tests can drive
+// the window rollover with a fake clock.
+internal interface IFixedWindowCounter
+{
+    /// <summary>
+    /// Atomically increments the counter for <paramref name="key"/> within the
+    /// current <paramref name="window"/>.
+    /// </summary>
+    /// <returns>
+    /// The count for <paramref name="key"/> after the increment, within the
+    /// current window. Counts never carry over across a window boundary.
+    /// </returns>
+    long Increment(string key, TimeSpan window);
+}

src/Cluckwork.Infrastructure/SharedState/ILease.cs

@@ -0,0 +1,57 @@
+namespace Cluckwork.Infrastructure.SharedState;
+
+// #543 — owned, renewable lease with compare-and-delete release (port for the
+// report-concurrency capacity cap, #545).
+//
+// This bounds how many heavy report queries an account runs at once; it is a
+// capacity protection, NOT a mutual-exclusion primitive for correctness. The
+// #271 job single-runner uses a Postgres advisory lock, deliberately not this
+// port — the resilient decorator falls back per-replica under a Redis outage,
+// which is fine for a capacity cap but would double-run a job. See
+// <see cref="ResilientLease"/>.
+//
+// A lease is held by exactly one <c>owner</c> token at a time, for a TTL. When
+// the TTL elapses without renewal the lease is free again and may be acquired
+// by anyone — a dead holder's lease is reclaimed by expiry, never by the
+// holder itself. Release is compare-and-delete: only the current holder may
+// release; an owner whose lease already expired and was re-acquired by someone
+// else must NOT release the new holder's lease. Time comes exclusively from
+// the injected <see cref="TimeProvider"/> — implementations must never read a
+// wall clock directly, so tests can drive expiry with a fake clock.
+internal interface ILease
+{
+    /// <summary>
+    /// Attempts to acquire the lease on <paramref name="key"/> for
+    /// <paramref name="ttl"/>.
+    /// </summary>
+    /// <returns>
+    /// <see langword="true"/> if the lease was free (unheld, or held by someone
+    /// whose TTL has elapsed) and is now held by <paramref name="owner"/>;
+    /// <see langword="false"/> if a live lease on the key is held by another
+    /// owner (the existing holder and TTL are left untouched).
+    /// </returns>
+    bool TryAcquire(string key, string owner, TimeSpan ttl);
+
+    /// <summary>
+    /// Attempts to extend the lease on <paramref name="key"/>.
+    /// </summary>
+    /// <returns>
+    /// <see langword="true"/> only if <paramref name="owner"/> still holds the
+    /// live lease, in which case the TTL is extended to
+    /// <paramref name="ttl"/> from now; <see langword="false"/> if the lease
+    /// has expired or is held by another owner (state left untouched).
+    /// </returns>
+    bool Renew(string key, string owner, TimeSpan ttl);
+
+    /// <summary>
+    /// Attempts to release the lease on <paramref name="key"/>.
+    /// </summary>
+    /// <returns>
+    /// <see langword="true"/> only if <paramref name="owner"/> still holds the
+    /// live lease, in which case the key becomes free; <see langword="false"/>
+    /// if the lease has expired or is held by another owner — in particular,
+    /// a previous owner whose lease expired and was re-acquired must NOT
+    /// release the new holder's lease.
+    /// </returns>
+    bool Release(string key, string owner);
+}