Skip to content

feat(ratelimit): distributed IP-keyed auth limiters (#544) - #558

Merged
mforce merged 3 commits into
mainfrom
feat/544-distributed-ip-rate-limiters
Aug 18, 2026
Merged

mforce merged 3 commits into
mainfrom
feat/544-distributed-ip-rate-limiters

Conversation

@mforce

@mforce mforce commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Closes #544. Epic #530 slice S3 (Phase 1.6 scale-out).

What

Moves the three IP-keyed fixed-window limiters (login / refresh / client-errors) off ASP.NET's per-process partitions onto the shared IFixedWindowCounter (#543, Redis-backed with in-process fallback + alarm), so N replicas enforce one combined per-IP budget instead of N.

Design

Tests

  • MultiInstanceRateLimitTests — two real dotnet Cluckwork.Api.dll processes + one real Redis Testcontainer prove one combined per-IP budget (split N across both instances, the N+1th 429s).
  • DistributedRateLimiterWiringTests — in-process (toggleable-primary stub): fallback+alarm, recovery-without-restart, pre-account-lookup limiting, and no attacker-supplied dimension in key or log.

Verified locally

Build 0/0; Domain 325 + Application 145 + Api.IntegrationTests 1256 all green. Mutation: forcing the limiter to bypass the shared counter turns the two-process combined-budget test RED (restored → green); dropping the policy prefix turns Refresh_has_its_own_bucket_independent_of_login RED.

Scope notes

mforce added 2 commits August 18, 2026 11:58
Move the login / refresh / client-errors IP-keyed fixed-window limiters off
ASP.NET's per-process partitions onto the shared IFixedWindowCounter (#543,
Redis-backed with in-process fallback + alarm), so N replicas enforce one
combined per-IP budget instead of N. Closes epic #530 slice S3.

Bridge mirrors the #338 precedent: the limiter + a public IRateLimiterPolicy<string>
live in Cluckwork.Infrastructure (same assembly as the internal IFixedWindowCounter
port); Api registers the policy. RateLimitKey moves Api -> Infrastructure so the
policy shares the one canonical IPv6-/64 derivation. The counter key carries the
policy name, so login/refresh/client-errors stay independent budgets while
login/step-up/change-password still share one (#273). Failure falls back to the
in-process counter and alarms (free, via the resilient decorator).

Tests: two real processes + one real Redis prove one combined per-IP budget
(MultiInstanceRateLimitTests); in-process wiring tests prove fallback+alarm,
recovery, pre-account-lookup limiting, and no attacker-supplied dimension in the
key or log (DistributedRateLimiterWiringTests). AGENTS.md #271 deploy-invariant
updated: #143 is no longer an in-process scaling blocker (#311 remains, -> #545).
…ter (#544 review)

Addresses the code-review hardening findings on the limiter's shared-counter use:

- Add IFixedWindowCounter.IncrementAsync returning FixedWindowResult(Count, Remaining).
  The limiter's async acquire path (the ASP.NET middleware hot path) now awaits it
  instead of calling the synchronous Increment, so an auth request never blocks a
  thread-pool thread on Redis I/O. The synchronous Increment is unchanged (still the
  sync AttemptAcquireCore path, off the middleware hot path).
- The Redis Lua script now returns PTTL alongside the count, so Retry-After on the async
  path comes from Redis's own server clock (Remaining) rather than a host-clock recompute
  that drifts under skew. In-process fallback returns windowEnd - now from the same
  injected clock it buckets on.
- Resilient decorator gains the async path (same fall-back-to-in-process + alarm).
- Nits: policy resolves its singletons inside the partition factory (cache-miss only);
  a comment on OnRejected => null names the test guarding the framework-fallback assumption.

Tests: async fallback (ResilientFallbackTests), Redis async contract returning a positive
Redis-clock Remaining, and direct limiter unit tests (Retry-After from the counter,
permitCount 0 no-consume / >1 throws). All 1263 integration tests green; the two-process
combined-budget test still goes red when the async path bypasses the shared counter.
@mforce

mforce commented Aug 18, 2026

Copy link
Copy Markdown
Owner Author

Local review round (2 Claude agents + deepseek) — addressed in 2e12fdc

Security-auditor: 0 Critical/High/Med/Low. Traced the counter key end-to-end — no attacker-supplied dimension (email/farm) reaches a key, Redis-key component, metric, or log; fallback alarmed; budgets independent; limit-before-lookup. One Info (framework OnRejected fallback assumption) — addressed with a guard comment naming the test that pins it.

Code-reviewer (hardening, no correctness/security defect): addressed at the #543 port level:

  • Limiter async acquire path called the synchronous IFixedWindowCounter.Increment → sync ScriptEvaluate, blocking a thread-pool thread on the auth hot path. Added IncrementAsync (async ScriptEvaluateAsync); the middleware hot path now awaits it. Sync Increment unchanged (off the hot path).
  • Retry-After computed on the host clock while Redis buckets on its own server clock. The Lua script now returns PTTL; the async path's Retry-After comes from the counter's own clock (in-process returns windowEnd - now).
  • Nits: DI resolved inside the partition factory (cache-miss only); comment on OnRejected => null.

Verification: build 0/0; 1263 integration tests green (+7 async). Mutation: the two-process/real-Redis combined-budget test goes RED when the async path bypasses the shared counter, GREEN on restore.

@codex review — please focus on the new async counter path (IncrementAsync + the Redis Lua {c, pttl} script atomicity/TTL), the sync/async split in DistributedIpFixedWindowRateLimiter, and confirm the IP-only keying / no-attacker-dimension invariant holds after the refactor.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e12fdc4c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@@ -29,10 +33,44 @@ internal sealed class RedisFixedWindowCounter(IConnectionMultiplexer redis, stri
if c == 1 then
redis.call('PEXPIRE', k, windowMs)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Expire the Redis key at the bucket boundary

When the first request arrives partway through a wall-clock-aligned window, this sets the TTL to a full window from that request, so the returned PTTL is not the time remaining in the current bucket. For example, a request fourteen minutes into a fifteen-minute bucket produces a Retry-After near fifteen minutes even though the counter resets in about one minute. Derive the expiry from the bucket boundary (for example, windowMs - (ms % windowMs)) before returning it as the remaining window.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8b21753. The script now expires the key at the bucket boundary — PEXPIRE k (windowMs - (ms % windowMs)) — so PTTL (and the async Retry-After) is the time left in the CURRENT window, not a full window from the first request. Added a guard (IncrementAsync_remaining_is_time_to_bucket_boundary_not_a_full_window) that phases the first increment 2000–2500ms into a 4s bucket and asserts Remaining < 3s; the prior positive-remaining test used a 30s window and only asserted <= window, so a full-window TTL slipped past it. All 82 rate-limit + shared-state tests green (build 0/0).

 codex P2)

The fixed window is wall-clock aligned (bucket = floor(ms/windowMs)), but the first
request in a bucket set PEXPIRE to a FULL window from now, so the returned PTTL —
and thus IncrementAsync's Retry-After — overstated the time left when the first
request landed partway into a bucket (14 min into a 15 min window returned ~15 min).
Enforcement was unaffected (the bucket key rolls at the boundary); only the advisory
Retry-After. Expire at the boundary instead: windowMs - (ms % windowMs).

Guard: RedisFixedWindowCounterContractTests phases the first increment deep into a
bucket and asserts Remaining is well under a full window (the prior positive-remaining
test used a 30s window and only asserted <= window, so a full-window TTL slipped past).
@mforce

mforce commented Aug 18, 2026

Copy link
Copy Markdown
Owner Author

@codex review — P2 (bucket-boundary TTL) is fixed in 8b21753: PEXPIRE k (windowMs - (ms % windowMs)), so PTTL/Retry-After reflects the current window, with a phased guard test. Please re-check the Lua script and the async Retry-After path.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 8b21753d5a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mforce
mforce merged commit ec14972 into main Aug 18, 2026
10 checks passed
@mforce
mforce deleted the feat/544-distributed-ip-rate-limiters branch August 18, 2026 20:02
mforce pushed a commit that referenced this pull request Sep 12, 2026
🤖 I have created a release *beep* *boop*
---


## [0.1.0](v0.0.4...v0.1.0)
(2026-09-12)


### ⚠ BREAKING CHANGES

* log in by farm code, with per-account email identity
([#532](#532)) (#564)

### Features

* **accounts:** add Account.Slug (farm code), suspend/reactivate,
list-accounts verb
([#531](#531))
([3fe9754](3fe9754))
* **accounts:** provision additional farms
([#581](#581))
([006f298](006f298))
* add Aspire local development AppHost
([#567](#567))
([2c9e6b9](2c9e6b9))
* add configurable worker sale allocation
([#619](#619))
([0955095](0955095))
* add searchable entity pickers
([#642](#642))
([60d2053](60d2053))
* **api:** provision-account takes an optional --timezone at creation
([#603](#603))
([#694](#694))
([a0aee39](a0aee39))
* **audit:** show the sales-line audit payload as a readable Details
column ([#745](#745))
([#749](#749))
([d26d389](d26d389))
* **auth:** add ApplicationUser.StepUpLogoutEpoch column
([#338](#338))
([#554](#554))
([18306ee](18306ee))
* certify over-cap simulation fixture bands
([#633](#633))
([a67b2e1](a67b2e1)),
closes [#627](#627)
* **cli:** rename-account verb to change a farm code
([#732](#732))
([#733](#733))
([4b70559](4b70559))
* **customers:** edit existing customer details
([#625](#625))
([#626](#626))
([062a55c](062a55c))
* **jobs:** single-runner leader gate for the durable job worker
([#271](#271))
([#555](#555))
([4148f9b](4148f9b))
* let owners change user email addresses
([#605](#605))
([842347b](842347b))
* log in by farm code, with per-account email identity
([#532](#532))
([#564](#564))
([68adb62](68adb62))
* **ratelimit:** distributed IP-keyed auth limiters
([#544](#544))
([#558](#558))
([ec14972](ec14972))
* **ratelimit:** distributed per-account report concurrency cap with
local-ceiling fallback
([#545](#545))
([#559](#559))
([1522e4e](1522e4e))
* **sales:** mark discounted lines, total the discount, and show it in
the Orders list ([#723](#723),
[#724](#724))
([#741](#741))
([1a07441](1a07441))
* **sales:** record list, old and new price in the order-line audit
payload ([#722](#722))
([#742](#742))
([97c866f](97c866f))
* **sales:** refuse an over-ceiling confirm from a Sales user
([#727](#727))
([#766](#766))
([8c0792a](8c0792a))
* **sales:** show what each order still owes, and filter the list to
unpaid ([#771](#771))
([ca59d68](ca59d68))
* **sales:** snapshot the list price on the order line and show the
discount ([#734](#734))
([cffed5e](cffed5e))
* **sales:** snapshot the product name and unit in the order-line audit
payload ([#747](#747))
([#748](#748))
([0481c06](0481c06))
* scope Worker reads to assigned flocks
([#388](#388))
([#611](#611))
([5884a9a](5884a9a))
* shared-state ports with Redis + in-process fallback
([#543](#543))
([#552](#552))
([f767fa9](f767fa9))
* suspend-account / reactivate-account operator verbs
([#534](#534))
([#573](#573))
([d0be26c](d0be26c))
* **tenancy:** write-side tenant guard + single-assignment TenantContext
([#546](#546))
([#561](#561))
([f371f1d](f371f1d))
* **web:** dashboard rework — capture-status tiles, 14-day trend, stock
as a stacked bar
([#654](#654))
([396ba23](396ba23))
* **web:** date-range filters on audit and expenses, and the stock lot
filter gets its bounded toolbar
([#666](#666),
[#667](#667),
[#653](#653))
([94b188f](94b188f))
* **web:** elevation hierarchy and sentence-case labels
([#651](#651),
[#652](#652))
([#661](#661))
([28db4c7](28db4c7))
* **web:** Expenses and Audit keep a clear-filters control while rows
are still showing
([#679](#679))
([#697](#697))
([b859982](b859982))
* **web:** expenses filters by a date range like its sibling screens
([#667](#667))
([f13858f](f13858f))
* **web:** key the farm brand palette per farm
([#586](#586))
([#600](#600))
([7183a43](7183a43))
* **web:** let operators forget remembered farms
([#598](#598))
([577d94e](577d94e))
* **web:** one-line provenance, bounded date filters, and empty states
that invite action
([#653](#653),
[#655](#655))
([#668](#668))
([80b53f4](80b53f4))
* **web:** prefill the farm code from ?farm= and remember it
([#535](#535))
([#588](#588))
([b7f5cc6](b7f5cc6))
* **web:** split authenticated routes into lazy chunks
([#620](#620))
([5089271](5089271))
* **web:** the audit log filters by a date range, and says which window
is empty ([#666](#666))
([63027e0](63027e0))
* **web:** typeset numbers as numbers and refresh the Help glossary
([#650](#650),
[#657](#657))
([af4fe11](af4fe11))


### Bug fixes

* **api:** order same-instant audit events by a durable monotonic key
([#700](#700))
([8fcf084](8fcf084))
* **api:** print the farm code from bootstrap-admin
([#589](#589))
([#594](#594))
([34032ac](34032ac))
* **audit:** show the price a line sold for, not its list price
([#759](#759))
([e6b37d0](e6b37d0))
* **audit:** store catalog enums by name and guard the add-item
transaction shape
([#751](#751))
([23609ff](23609ff))
* **auth:** reject invalid account claims
([#622](#622))
([8d6c7fe](8d6c7fe))
* **auth:** require step-up for durable user access
([#360](#360))
([#607](#607))
([f767dce](f767dce))
* **ci:** bound the npm audit calls and give the web job room to finish
([#686](#686))
([153b7a8](153b7a8))
* **ci:** escalate the audit bound to SIGKILL, so it actually bounds
([#686](#686))
([a0c8f4e](a0c8f4e))
* **ci:** fail closed on invalid vulnerability config
([#621](#621))
([1690db8](1690db8))
* **ci:** lockfix covers the two AppHost lock files, derived from the
sln
([efb05e6](efb05e6))
* **ci:** lockfix covers the two AppHost lock files, derived from the
sln
([8986d77](8986d77))
* **ci:** remove invalid XML comment from nuget.lockfix.config
([#541](#541))
([5f1bc0a](5f1bc0a))
* **ci:** the advisory vuln gate no longer blocks on an unusable report
([#686](#686))
([aaf6934](aaf6934))
* **ci:** the advisory vuln gate no longer blocks on an unusable report
([#686](#686))
([64f1f53](64f1f53))
* **i18n:** tl help text names the saleable flag and unit-system setting
what their labels call them
([#688](#688))
([#696](#696))
([bfd24d7](bfd24d7))
* **infra:** AccountId must be a non-nullable Guid or both tenant write
layers refuse ([#673](#673))
([#695](#695))
([2470c4e](2470c4e))
* require step-up for flock scope changes
([#609](#609))
([4151f89](4151f89))
* **sales:** keep a line's discount markers agreeing while its price is
edited ([#752](#752))
([#753](#753))
([c159b4b](c159b4b))
* **sales:** say which kind of missing list price a line has
([#774](#774))
([489180e](489180e))
* scope legacy logout to selected farm
([#624](#624))
([fae8d82](fae8d82))
* **seed:** drain the daily-entry lock sweep so deep simulation fixtures
validate ([#644](#644))
([730fa23](730fa23)),
closes [#638](#638)
* **tenancy:** AccountId is a concurrency token, so the database refuses
a detached cross-tenant write
([#562](#562))
([4d1dfa3](4d1dfa3))
* **tenancy:** AspNetUserRoles carries a tenant column, so a role write
naming another farm's user is refused
([#670](#670))
([fc0552a](fc0552a))
* **tests:** bump the image-pin allow-list counts for the AppHost
LocalPorts tests
([#593](#593))
([58d3056](58d3056))
* **tests:** the OTLP collector survives a lost port race and ignores
traffic that is not an export
([#672](#672),
[#676](#676))
([#677](#677))
([965c737](965c737))
* **web:** a scoped audit view filtered to nothing names both the record
and the range ([#666](#666))
([41bbfe1](41bbfe1))
* **web:** an abandoned dialog attempt's success no longer hijacks the
replacement on Customers, Daily Entry, Flocks, Grades and Products
([#703](#703))
([#705](#705))
([85605db](85605db))
* **web:** an abandoned dialog attempt's success no longer hijacks the
replacement on Inventory, Expenses, History and Stock
([#703](#703))
([#706](#706))
([60a4997](60a4997))
* **web:** an abandoned edit's success no longer hijacks the dialog that
replaced it on Users
([#703](#703))
([#710](#710))
([778faab](778faab))
* **web:** an abandoned order attempt's success no longer hijacks the
dialog that replaced it
([#702](#702))
([522c699](522c699))
* **web:** capture screens open on the flock you last used, and
assigning one no longer guesses
([#646](#646))
([#699](#699))
([7f8f317](7f8f317))
* **web:** constrain dialog session helpers to declared scopes
([#715](#715))
([389e3c8](389e3c8))
* **web:** date validation gets one boundary table instead of one case
per review round
([#666](#666))
([215f830](215f830))
* **web:** keep a paged window and an item panel on the user's newest
intent ([#645](#645))
([d81bccf](d81bccf))
* **web:** keep Sales order panels closed after pending writes
([#711](#711))
([f0f7492](f0f7492))
* **web:** keep Sales panels closed after pending Open reads
([#716](#716))
([620411f](620411f))
* **web:** make login take the cross-tab cookie lock so a racing refresh
cannot restore the wrong session
([#648](#648))
([ff18beb](ff18beb))
* **web:** make the entity picker read as a search field and focus it on
open ([#736](#736))
([66ef667](66ef667)),
closes [#735](#735)
* **web:** page truncated customer and movement tables with usePagedList
([7cfe4d6](7cfe4d6))
* **web:** reconcile Sales line edits with refreshed orders
([#717](#717))
([d7dd2c9](d7dd2c9))
* **web:** the audit date filter accepts low-numbered years, and its
empty state covers every narrowing
([#666](#666))
([af52d25](af52d25))
* **web:** the audit date filter rejects impossible dates, and its
history guard actually guards
([#666](#666))
([8d51846](8d51846))
* **web:** the expense range bounds are not capped at today, which the
month-end default exceeds
([#667](#667))
([7e01864](7e01864))
* **web:** the help text calls the expiry field what the field calls
itself ([#666](#666))
([2fd1f3c](2fd1f3c))
* **web:** the stock lot date range sits in the bounded toolbar
([#653](#653))
([43dec5e](43dec5e))


### Refactoring

* **web:** extract SalesPage's dialog-write wrapper into a shared
useDialogAction hook
([#703](#703))
([#704](#704))
([60ee9d9](60ee9d9))


### Documentation

* add k6 preparation steps to the dev-database fixture runbook
([#643](#643))
([a4f1f09](a4f1f09))
* add runbook for loading the simulation fixture into a dev database
([#639](#639))
([2d143b8](2d143b8))
* **agents:** a PR closes its issue from the body, not the title
([#744](#744))
([39be13c](39be13c))
* **agents:** drop the commit and push gate, and require screenshots on
UI changes ([#757](#757))
([6225172](6225172))
* **agents:** find guards by grepping registry readers; amend issues a
PR overtakes ([#580](#580))
([fe3fde8](fe3fde8))
* **agents:** the Playwright specs have been in CI since 2026-08-08
([#768](#768))
([68ee612](68ee612))
* **aspire:** record the second local database and pin the AppHost
dashboard ports ([#623](#623))
([713b941](713b941))
* compress AGENTS.md to one paragraph per rule, and draw the two orders
that matter ([#551](#551))
([997ae8a](997ae8a))
* item 7 names each screen's actual initial filter value
([#666](#666))
([70a53d8](70a53d8))
* multi-farm tenancy decision record and AGENTS/GLOSSARY sync
([#537](#537))
([#601](#601))
([2c34771](2c34771))
* name the scoped filtered-empty key and state the
[#653](#653) relationship
plainly ([#666](#666))
([0e93dac](0e93dac))
* note that a PackageReference in Directory.Build.props is invisible to
the dependency graph
([4845724](4845724))
* **plans:** commit the
[#722](#722) and
[#745](#745) design records
([#754](#754))
([c942fcd](c942fcd))
* record [#579](#579) as
won't-fix — suspension is immediate for use, not issuance
([#582](#582))
([7a3be40](7a3be40))
* record the [#508](#508)
audit ordering key and the tracked-file guard lesson
([#701](#701))
([08964e9](08964e9))
* **runbooks:** add procedure to rename the default farm's code after
upgrade ([#731](#731))
([2f6e242](2f6e242))
* screenshots of the running SPA in the README
([#550](#550))
([711488a](711488a))
* **sim:** commit the dashboard screenshot, capture the palette matrix,
and record the
[#651](https://github.com/mforce/cluckwork/issues/651)/[#652](https://github.com/mforce/cluckwork/issues/652)
conventions ([#660](#660),
[#662](#662),
[#663](#663),
[#664](#664))
([#665](#665))
([930ea30](930ea30))
* specify searchable entity picker
([#641](#641))
([91d4300](91d4300))
* split the README into audience-scoped docs and adopt repo-template
scaffolding ([#548](#548))
([b3f3fcf](b3f3fcf))
* surface Aspire local development workflow
([#568](#568))
([a343baa](a343baa))
* **web:** record the per-screen idempotency-key policies and runWrite's
refresh contract
([#703](#703))
([#707](#707))
([8bee651](8bee651))
* **web:** the date-cap help text covers every stocked item, not only
feed ([#666](#666),
[#667](#667))
([c8433c5](c8433c5))
* **web:** the help text claims only what is true of recording, and says
nothing about filter caps
([#666](#666),
[#667](#667))
([e2f63d1](e2f63d1))
* **web:** the help text describes the date-range filters that shipped
([#666](#666),
[#667](#667))
([c3275b7](c3275b7))
* **web:** the help text stops describing a cap the filters no longer
have ([#666](#666),
[#667](#667))
([49654cd](49654cd))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: cluckwork-lockfix[bot] <309265648+cluckwork-lockfix[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Distributed IP-keyed auth rate limiters with in-process fallback

1 participant