Skip to content

Batch 2 — Resilience & Startup (#2 graceful degradation, #3 Cosmos lazy init, #8 Redis lazy connect, #9 Redis expiry DB) - #13

Merged
jzills merged 14 commits into
developfrom
fix/batch-2-resilience-startup
Jul 8, 2026
Merged

jzills merged 14 commits into
developfrom
fix/batch-2-resilience-startup

Conversation

@jzills

@jzills jzills commented Jul 8, 2026 •

Copy link
Copy Markdown
Owner

Batch 2 — Resilience & Startup

Second of four themed batches from a full repository recon (Batch 1, correctness & key hygiene, merged as #12). This batch makes ActionCache tolerate cache-backend failures at runtime (fail-open by default) and at startup (lazy, non-blocking initialization), and targets Redis expiry cleanup at the configured database.

What's fixed

#2 — Graceful degradation (fail-open by default, opt-in fail-closed)
A new public ResilientActionCache decorator now wraps every backend cache: a transient Redis/SQL/Cosmos error is logged (Warning) and the operation degrades to a cache miss (reads) or a no-op (writes/eviction/refresh) so the request still succeeds without caching. It's applied centrally in the two abstract filter factories (MVC + Minimal API) at the single point caches enter the handler — IActionCacheFactory.Create and the backend factories stay untouched. Opt in to propagate backend errors instead:

builder.Services.AddActionCache(options =>
{
    options.UseRedisCache(...);
    options.FailClosed();
});

#3 — Cosmos: initialize once, lazily
AddActionCacheAzureCosmos used to run CreateDatabaseIfNotExistsAsync + CreateContainerIfNotExistsAsync sync-over-async on every request scope. Setup now runs once via a singleton AsyncLazy<Container>, awaited on first cache operation — no per-request round-trips, no thread-pool blocking, no startup blocking. The lazy holder re-initializes if the first attempt faults, so a transient outage at first use doesn't permanently disable caching for the process.

#8 — Redis: lazy, non-blocking connect
ConnectionMultiplexer.Connect(...) was eager at DI registration, so a Redis outage at startup stopped the app from booting. It's now a lazy singleton factory with AbortOnConnectFail=false — the app boots, the multiplexer reconnects in the background, and failed operations degrade via the resilience decorator.

#9 — Redis: expiry channel from configured DB + docs
RedisExpiryService subscribed to the hardcoded __keyevent@0__:expired, so a connection on a non-zero database silently missed all expiry events (orphaned sorted-set index members). It now derives the DB index from the connection (__keyevent@{db}__:expired) and wraps the subscribe so a Redis outage logs instead of crashing the host. The README documents the notify-keyspace-events Ex server requirement (and that the index self-heals lazily on access without it).

⚠️ Breaking changes (pre-1.0, intentional — flagged in commit bodies)

  • The two abstract filter-factory constructors gain a ResilientCacheDecorator parameter (DI-resolved).
  • AzureCosmosActionCache / AzureCosmosActionCacheFactory constructors now take AsyncLazy<Container>; AzureCosmosActionCacheProvider.CreateAsync → CreateContainerAsync returning a Container.
  • Redis AbortOnConnectFail now defaults to false (the app no longer fails to start when Redis is unreachable at boot).
  • RedisExpiryService constructor gains an ILogger<RedisExpiryService> parameter.

Testing

  • Unit: 427/427 on net8.0 and net10.0; Release build 0 warnings / 0 errors.
  • New tests: decorator fail-open/fail-closed per operation (+ accurate fail-closed Error logging), AsyncLazy once-only and retry-after-fault, lazy multiplexer registration (factory, not eager instance), and expiry-channel DB derivation.
  • Integration suite requires live Redis / SQL Server / Cosmos and can't run in the dev sandbox — please confirm the CI integration job is green.

Whole-branch review follow-ups (now addressed in this PR)

The follow-ups the review flagged as non-blocking have since been folded into this PR:

  • Broadened fail-closed test coverage to all six operations, plus a test asserting the fail-closed path logs at Error and rethrows.
  • Redis expiry subscribe now retries until it succeeds instead of giving up for the process lifetime (once subscribed, StackExchange.Redis re-establishes across reconnects); cancellation during the wait is a clean stop.
  • DRY'd the two byte-identical abstract filter factories into ActionCacheFilterAbstractFactoryBase<TFilter> (subclasses provide only CreateFilter), and fixed the double-enumeration so each backend factory's Create runs once (was twice: null-check + wrap).
  • Polish: ResilientActionCache.GetNamespace() returns the cached namespace; a <remarks> documents the intentionally-broad catch; hardened the AsyncLazy once-only test to exercise real thread contention.

🤖 Generated with Claude Code

jzills added 14 commits July 7, 2026 15:52
…ries

Wraps every backend cache in a fail-open decorator at the point caches enter
the handler. BREAKING (pre-1.0): the two abstract filter factory constructors
gain a ResilientCacheDecorator parameter.

Also updates ActionCacheFilterFactoriesTests.cs, a third direct consumer of
both abstract-factory constructors not listed in the task brief's file list,
so the unit suite compiles against the new constructor signature.
…ner>

Removes per-request sync-over-async CreateIfNotExists. BREAKING (pre-1.0):
AzureCosmosActionCache and AzureCosmosActionCacheFactory now take
AsyncLazy<Container>; AzureCosmosActionCacheProvider.CreateAsync becomes
CreateContainerAsync returning a Container.
The multiplexer now connects on first use via a factory registration instead
of eagerly at DI registration, so a Redis outage at startup no longer stops the
app from booting. BREAKING (pre-1.0): AbortOnConnectFail defaults to false.
…ibe failure

Subscribes to __keyevent@{db}__:expired for the connection's configured database
instead of hardcoded db 0, so non-zero-DB deployments receive expiry events. The
subscribe is wrapped so a Redis outage logs instead of crashing the host. Documents
the notify-keyspace-events Ex requirement.

BREAKING (pre-1.0): RedisExpiryService constructor now requires an
ILogger<RedisExpiryService> parameter.
- GetNamespace() returns the cached namespace instead of re-delegating
- document the intentionally-broad catch (incl. OperationCanceledException) via <remarks>
- add fail-closed rethrow tests for GetKeysAsync/RemoveAsync(key)/RefreshAsync
- drop an unused using in the tests

Addresses non-blocking follow-ups from the Batch 2 whole-branch review.
…test

Dispatch the 25 concurrent Value accesses via Task.Run so they actually race
on the lock instead of being enumerated synchronously on the test thread.
…che instances once

The MVC and Minimal API abstract filter factories were byte-identical except for
the concrete filter they build. Hoist the shared namespace-resolution and
cache-instance creation into ActionCacheFilterAbstractFactoryBase<TFilter>;
subclasses now provide only CreateFilter. AddCacheInstances materializes the
factory results once so each factory's Create runs a single time (was twice: once
for the null-check, once for the wrap). Adds a regression test per factory.
If Redis is unreachable when the host starts, the expiry listener now logs and
retries the subscribe on an interval instead of giving up for the process
lifetime; once subscribed, StackExchange.Redis re-establishes the subscription
across reconnects. Cancellation during the wait is treated as a clean stop.
…ntial backoff

A Redis outage is a tolerated, fail-open condition, so RedisExpiryService now
logs subscribe failures at Warning (matching ResilientActionCache) instead of
Error, and backs off exponentially (InitialRetryDelay 1s, doubling to a
MaxRetryDelay 30s cap) instead of retrying on a fixed 30s loop. Since the loop
runs only until the first successful subscribe, this keeps early retries quick
while preventing a sustained outage from flooding the logs.
@jzills
jzills merged commit c306e08 into develop Jul 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant