Repository navigation
feat(ratelimit): distributed IP-keyed auth limiters (#544) - #558
Conversation
Move the login / refresh / client-errors IP-keyed fixed-window limiters off ASP.NET's per-process partitions onto the shared IFixedWindowCounter (#543, Redis-backed with in-process fallback + alarm), so N replicas enforce one combined per-IP budget instead of N. Closes epic #530 slice S3. Bridge mirrors the #338 precedent: the limiter + a public IRateLimiterPolicy<string> live in Cluckwork.Infrastructure (same assembly as the internal IFixedWindowCounter port); Api registers the policy. RateLimitKey moves Api -> Infrastructure so the policy shares the one canonical IPv6-/64 derivation. The counter key carries the policy name, so login/refresh/client-errors stay independent budgets while login/step-up/change-password still share one (#273). Failure falls back to the in-process counter and alarms (free, via the resilient decorator). Tests: two real processes + one real Redis prove one combined per-IP budget (MultiInstanceRateLimitTests); in-process wiring tests prove fallback+alarm, recovery, pre-account-lookup limiting, and no attacker-supplied dimension in the key or log (DistributedRateLimiterWiringTests). AGENTS.md #271 deploy-invariant updated: #143 is no longer an in-process scaling blocker (#311 remains, -> #545).
…ter (#544 review) Addresses the code-review hardening findings on the limiter's shared-counter use: - Add IFixedWindowCounter.IncrementAsync returning FixedWindowResult(Count, Remaining). The limiter's async acquire path (the ASP.NET middleware hot path) now awaits it instead of calling the synchronous Increment, so an auth request never blocks a thread-pool thread on Redis I/O. The synchronous Increment is unchanged (still the sync AttemptAcquireCore path, off the middleware hot path). - The Redis Lua script now returns PTTL alongside the count, so Retry-After on the async path comes from Redis's own server clock (Remaining) rather than a host-clock recompute that drifts under skew. In-process fallback returns windowEnd - now from the same injected clock it buckets on. - Resilient decorator gains the async path (same fall-back-to-in-process + alarm). - Nits: policy resolves its singletons inside the partition factory (cache-miss only); a comment on OnRejected => null names the test guarding the framework-fallback assumption. Tests: async fallback (ResilientFallbackTests), Redis async contract returning a positive Redis-clock Remaining, and direct limiter unit tests (Retry-After from the counter, permitCount 0 no-consume / >1 throws). All 1263 integration tests green; the two-process combined-budget test still goes red when the async path bypasses the shared counter.
Local review round (2 Claude agents + deepseek) — addressed in 2e12fdcSecurity-auditor: 0 Critical/High/Med/Low. Traced the counter key end-to-end — no attacker-supplied dimension (email/farm) reaches a key, Redis-key component, metric, or log; fallback alarmed; budgets independent; limit-before-lookup. One Info (framework Code-reviewer (hardening, no correctness/security defect): addressed at the #543 port level:
Verification: build 0/0; 1263 integration tests green (+7 async). Mutation: the two-process/real-Redis combined-budget test goes RED when the async path bypasses the shared counter, GREEN on restore. @codex review — please focus on the new async counter path ( |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e12fdc4c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -29,10 +33,44 @@ internal sealed class RedisFixedWindowCounter(IConnectionMultiplexer redis, stri | |||
| if c == 1 then | |||
| redis.call('PEXPIRE', k, windowMs) | |||
There was a problem hiding this comment.
Expire the Redis key at the bucket boundary
When the first request arrives partway through a wall-clock-aligned window, this sets the TTL to a full window from that request, so the returned PTTL is not the time remaining in the current bucket. For example, a request fourteen minutes into a fifteen-minute bucket produces a Retry-After near fifteen minutes even though the counter resets in about one minute. Derive the expiry from the bucket boundary (for example, windowMs - (ms % windowMs)) before returning it as the remaining window.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 8b21753. The script now expires the key at the bucket boundary — PEXPIRE k (windowMs - (ms % windowMs)) — so PTTL (and the async Retry-After) is the time left in the CURRENT window, not a full window from the first request. Added a guard (IncrementAsync_remaining_is_time_to_bucket_boundary_not_a_full_window) that phases the first increment 2000–2500ms into a 4s bucket and asserts Remaining < 3s; the prior positive-remaining test used a 30s window and only asserted <= window, so a full-window TTL slipped past it. All 82 rate-limit + shared-state tests green (build 0/0).
codex P2) The fixed window is wall-clock aligned (bucket = floor(ms/windowMs)), but the first request in a bucket set PEXPIRE to a FULL window from now, so the returned PTTL — and thus IncrementAsync's Retry-After — overstated the time left when the first request landed partway into a bucket (14 min into a 15 min window returned ~15 min). Enforcement was unaffected (the bucket key rolls at the boundary); only the advisory Retry-After. Expire at the boundary instead: windowMs - (ms % windowMs). Guard: RedisFixedWindowCounterContractTests phases the first increment deep into a bucket and asserts Remaining is well under a full window (the prior positive-remaining test used a 30s window and only asserted <= window, so a full-window TTL slipped past).
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🤖 I have created a release *beep* *boop* --- ## [0.1.0](v0.0.4...v0.1.0) (2026-09-12) ### ⚠ BREAKING CHANGES * log in by farm code, with per-account email identity ([#532](#532)) (#564) ### Features * **accounts:** add Account.Slug (farm code), suspend/reactivate, list-accounts verb ([#531](#531)) ([3fe9754](3fe9754)) * **accounts:** provision additional farms ([#581](#581)) ([006f298](006f298)) * add Aspire local development AppHost ([#567](#567)) ([2c9e6b9](2c9e6b9)) * add configurable worker sale allocation ([#619](#619)) ([0955095](0955095)) * add searchable entity pickers ([#642](#642)) ([60d2053](60d2053)) * **api:** provision-account takes an optional --timezone at creation ([#603](#603)) ([#694](#694)) ([a0aee39](a0aee39)) * **audit:** show the sales-line audit payload as a readable Details column ([#745](#745)) ([#749](#749)) ([d26d389](d26d389)) * **auth:** add ApplicationUser.StepUpLogoutEpoch column ([#338](#338)) ([#554](#554)) ([18306ee](18306ee)) * certify over-cap simulation fixture bands ([#633](#633)) ([a67b2e1](a67b2e1)), closes [#627](#627) * **cli:** rename-account verb to change a farm code ([#732](#732)) ([#733](#733)) ([4b70559](4b70559)) * **customers:** edit existing customer details ([#625](#625)) ([#626](#626)) ([062a55c](062a55c)) * **jobs:** single-runner leader gate for the durable job worker ([#271](#271)) ([#555](#555)) ([4148f9b](4148f9b)) * let owners change user email addresses ([#605](#605)) ([842347b](842347b)) * log in by farm code, with per-account email identity ([#532](#532)) ([#564](#564)) ([68adb62](68adb62)) * **ratelimit:** distributed IP-keyed auth limiters ([#544](#544)) ([#558](#558)) ([ec14972](ec14972)) * **ratelimit:** distributed per-account report concurrency cap with local-ceiling fallback ([#545](#545)) ([#559](#559)) ([1522e4e](1522e4e)) * **sales:** mark discounted lines, total the discount, and show it in the Orders list ([#723](#723), [#724](#724)) ([#741](#741)) ([1a07441](1a07441)) * **sales:** record list, old and new price in the order-line audit payload ([#722](#722)) ([#742](#742)) ([97c866f](97c866f)) * **sales:** refuse an over-ceiling confirm from a Sales user ([#727](#727)) ([#766](#766)) ([8c0792a](8c0792a)) * **sales:** show what each order still owes, and filter the list to unpaid ([#771](#771)) ([ca59d68](ca59d68)) * **sales:** snapshot the list price on the order line and show the discount ([#734](#734)) ([cffed5e](cffed5e)) * **sales:** snapshot the product name and unit in the order-line audit payload ([#747](#747)) ([#748](#748)) ([0481c06](0481c06)) * scope Worker reads to assigned flocks ([#388](#388)) ([#611](#611)) ([5884a9a](5884a9a)) * shared-state ports with Redis + in-process fallback ([#543](#543)) ([#552](#552)) ([f767fa9](f767fa9)) * suspend-account / reactivate-account operator verbs ([#534](#534)) ([#573](#573)) ([d0be26c](d0be26c)) * **tenancy:** write-side tenant guard + single-assignment TenantContext ([#546](#546)) ([#561](#561)) ([f371f1d](f371f1d)) * **web:** dashboard rework — capture-status tiles, 14-day trend, stock as a stacked bar ([#654](#654)) ([396ba23](396ba23)) * **web:** date-range filters on audit and expenses, and the stock lot filter gets its bounded toolbar ([#666](#666), [#667](#667), [#653](#653)) ([94b188f](94b188f)) * **web:** elevation hierarchy and sentence-case labels ([#651](#651), [#652](#652)) ([#661](#661)) ([28db4c7](28db4c7)) * **web:** Expenses and Audit keep a clear-filters control while rows are still showing ([#679](#679)) ([#697](#697)) ([b859982](b859982)) * **web:** expenses filters by a date range like its sibling screens ([#667](#667)) ([f13858f](f13858f)) * **web:** key the farm brand palette per farm ([#586](#586)) ([#600](#600)) ([7183a43](7183a43)) * **web:** let operators forget remembered farms ([#598](#598)) ([577d94e](577d94e)) * **web:** one-line provenance, bounded date filters, and empty states that invite action ([#653](#653), [#655](#655)) ([#668](#668)) ([80b53f4](80b53f4)) * **web:** prefill the farm code from ?farm= and remember it ([#535](#535)) ([#588](#588)) ([b7f5cc6](b7f5cc6)) * **web:** split authenticated routes into lazy chunks ([#620](#620)) ([5089271](5089271)) * **web:** the audit log filters by a date range, and says which window is empty ([#666](#666)) ([63027e0](63027e0)) * **web:** typeset numbers as numbers and refresh the Help glossary ([#650](#650), [#657](#657)) ([af4fe11](af4fe11)) ### Bug fixes * **api:** order same-instant audit events by a durable monotonic key ([#700](#700)) ([8fcf084](8fcf084)) * **api:** print the farm code from bootstrap-admin ([#589](#589)) ([#594](#594)) ([34032ac](34032ac)) * **audit:** show the price a line sold for, not its list price ([#759](#759)) ([e6b37d0](e6b37d0)) * **audit:** store catalog enums by name and guard the add-item transaction shape ([#751](#751)) ([23609ff](23609ff)) * **auth:** reject invalid account claims ([#622](#622)) ([8d6c7fe](8d6c7fe)) * **auth:** require step-up for durable user access ([#360](#360)) ([#607](#607)) ([f767dce](f767dce)) * **ci:** bound the npm audit calls and give the web job room to finish ([#686](#686)) ([153b7a8](153b7a8)) * **ci:** escalate the audit bound to SIGKILL, so it actually bounds ([#686](#686)) ([a0c8f4e](a0c8f4e)) * **ci:** fail closed on invalid vulnerability config ([#621](#621)) ([1690db8](1690db8)) * **ci:** lockfix covers the two AppHost lock files, derived from the sln ([efb05e6](efb05e6)) * **ci:** lockfix covers the two AppHost lock files, derived from the sln ([8986d77](8986d77)) * **ci:** remove invalid XML comment from nuget.lockfix.config ([#541](#541)) ([5f1bc0a](5f1bc0a)) * **ci:** the advisory vuln gate no longer blocks on an unusable report ([#686](#686)) ([aaf6934](aaf6934)) * **ci:** the advisory vuln gate no longer blocks on an unusable report ([#686](#686)) ([64f1f53](64f1f53)) * **i18n:** tl help text names the saleable flag and unit-system setting what their labels call them ([#688](#688)) ([#696](#696)) ([bfd24d7](bfd24d7)) * **infra:** AccountId must be a non-nullable Guid or both tenant write layers refuse ([#673](#673)) ([#695](#695)) ([2470c4e](2470c4e)) * require step-up for flock scope changes ([#609](#609)) ([4151f89](4151f89)) * **sales:** keep a line's discount markers agreeing while its price is edited ([#752](#752)) ([#753](#753)) ([c159b4b](c159b4b)) * **sales:** say which kind of missing list price a line has ([#774](#774)) ([489180e](489180e)) * scope legacy logout to selected farm ([#624](#624)) ([fae8d82](fae8d82)) * **seed:** drain the daily-entry lock sweep so deep simulation fixtures validate ([#644](#644)) ([730fa23](730fa23)), closes [#638](#638) * **tenancy:** AccountId is a concurrency token, so the database refuses a detached cross-tenant write ([#562](#562)) ([4d1dfa3](4d1dfa3)) * **tenancy:** AspNetUserRoles carries a tenant column, so a role write naming another farm's user is refused ([#670](#670)) ([fc0552a](fc0552a)) * **tests:** bump the image-pin allow-list counts for the AppHost LocalPorts tests ([#593](#593)) ([58d3056](58d3056)) * **tests:** the OTLP collector survives a lost port race and ignores traffic that is not an export ([#672](#672), [#676](#676)) ([#677](#677)) ([965c737](965c737)) * **web:** a scoped audit view filtered to nothing names both the record and the range ([#666](#666)) ([41bbfe1](41bbfe1)) * **web:** an abandoned dialog attempt's success no longer hijacks the replacement on Customers, Daily Entry, Flocks, Grades and Products ([#703](#703)) ([#705](#705)) ([85605db](85605db)) * **web:** an abandoned dialog attempt's success no longer hijacks the replacement on Inventory, Expenses, History and Stock ([#703](#703)) ([#706](#706)) ([60a4997](60a4997)) * **web:** an abandoned edit's success no longer hijacks the dialog that replaced it on Users ([#703](#703)) ([#710](#710)) ([778faab](778faab)) * **web:** an abandoned order attempt's success no longer hijacks the dialog that replaced it ([#702](#702)) ([522c699](522c699)) * **web:** capture screens open on the flock you last used, and assigning one no longer guesses ([#646](#646)) ([#699](#699)) ([7f8f317](7f8f317)) * **web:** constrain dialog session helpers to declared scopes ([#715](#715)) ([389e3c8](389e3c8)) * **web:** date validation gets one boundary table instead of one case per review round ([#666](#666)) ([215f830](215f830)) * **web:** keep a paged window and an item panel on the user's newest intent ([#645](#645)) ([d81bccf](d81bccf)) * **web:** keep Sales order panels closed after pending writes ([#711](#711)) ([f0f7492](f0f7492)) * **web:** keep Sales panels closed after pending Open reads ([#716](#716)) ([620411f](620411f)) * **web:** make login take the cross-tab cookie lock so a racing refresh cannot restore the wrong session ([#648](#648)) ([ff18beb](ff18beb)) * **web:** make the entity picker read as a search field and focus it on open ([#736](#736)) ([66ef667](66ef667)), closes [#735](#735) * **web:** page truncated customer and movement tables with usePagedList ([7cfe4d6](7cfe4d6)) * **web:** reconcile Sales line edits with refreshed orders ([#717](#717)) ([d7dd2c9](d7dd2c9)) * **web:** the audit date filter accepts low-numbered years, and its empty state covers every narrowing ([#666](#666)) ([af52d25](af52d25)) * **web:** the audit date filter rejects impossible dates, and its history guard actually guards ([#666](#666)) ([8d51846](8d51846)) * **web:** the expense range bounds are not capped at today, which the month-end default exceeds ([#667](#667)) ([7e01864](7e01864)) * **web:** the help text calls the expiry field what the field calls itself ([#666](#666)) ([2fd1f3c](2fd1f3c)) * **web:** the stock lot date range sits in the bounded toolbar ([#653](#653)) ([43dec5e](43dec5e)) ### Refactoring * **web:** extract SalesPage's dialog-write wrapper into a shared useDialogAction hook ([#703](#703)) ([#704](#704)) ([60ee9d9](60ee9d9)) ### Documentation * add k6 preparation steps to the dev-database fixture runbook ([#643](#643)) ([a4f1f09](a4f1f09)) * add runbook for loading the simulation fixture into a dev database ([#639](#639)) ([2d143b8](2d143b8)) * **agents:** a PR closes its issue from the body, not the title ([#744](#744)) ([39be13c](39be13c)) * **agents:** drop the commit and push gate, and require screenshots on UI changes ([#757](#757)) ([6225172](6225172)) * **agents:** find guards by grepping registry readers; amend issues a PR overtakes ([#580](#580)) ([fe3fde8](fe3fde8)) * **agents:** the Playwright specs have been in CI since 2026-08-08 ([#768](#768)) ([68ee612](68ee612)) * **aspire:** record the second local database and pin the AppHost dashboard ports ([#623](#623)) ([713b941](713b941)) * compress AGENTS.md to one paragraph per rule, and draw the two orders that matter ([#551](#551)) ([997ae8a](997ae8a)) * item 7 names each screen's actual initial filter value ([#666](#666)) ([70a53d8](70a53d8)) * multi-farm tenancy decision record and AGENTS/GLOSSARY sync ([#537](#537)) ([#601](#601)) ([2c34771](2c34771)) * name the scoped filtered-empty key and state the [#653](#653) relationship plainly ([#666](#666)) ([0e93dac](0e93dac)) * note that a PackageReference in Directory.Build.props is invisible to the dependency graph ([4845724](4845724)) * **plans:** commit the [#722](#722) and [#745](#745) design records ([#754](#754)) ([c942fcd](c942fcd)) * record [#579](#579) as won't-fix — suspension is immediate for use, not issuance ([#582](#582)) ([7a3be40](7a3be40)) * record the [#508](#508) audit ordering key and the tracked-file guard lesson ([#701](#701)) ([08964e9](08964e9)) * **runbooks:** add procedure to rename the default farm's code after upgrade ([#731](#731)) ([2f6e242](2f6e242)) * screenshots of the running SPA in the README ([#550](#550)) ([711488a](711488a)) * **sim:** commit the dashboard screenshot, capture the palette matrix, and record the [#651](https://github.com/mforce/cluckwork/issues/651)/[#652](https://github.com/mforce/cluckwork/issues/652) conventions ([#660](#660), [#662](#662), [#663](#663), [#664](#664)) ([#665](#665)) ([930ea30](930ea30)) * specify searchable entity picker ([#641](#641)) ([91d4300](91d4300)) * split the README into audience-scoped docs and adopt repo-template scaffolding ([#548](#548)) ([b3f3fcf](b3f3fcf)) * surface Aspire local development workflow ([#568](#568)) ([a343baa](a343baa)) * **web:** record the per-screen idempotency-key policies and runWrite's refresh contract ([#703](#703)) ([#707](#707)) ([8bee651](8bee651)) * **web:** the date-cap help text covers every stocked item, not only feed ([#666](#666), [#667](#667)) ([c8433c5](c8433c5)) * **web:** the help text claims only what is true of recording, and says nothing about filter caps ([#666](#666), [#667](#667)) ([e2f63d1](e2f63d1)) * **web:** the help text describes the date-range filters that shipped ([#666](#666), [#667](#667)) ([c3275b7](c3275b7)) * **web:** the help text stops describing a cap the filters no longer have ([#666](#666), [#667](#667)) ([49654cd](49654cd)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: cluckwork-lockfix[bot] <309265648+cluckwork-lockfix[bot]@users.noreply.github.com>
Closes #544. Epic #530 slice S3 (Phase 1.6 scale-out).
What
Moves the three IP-keyed fixed-window limiters (
login/refresh/client-errors) off ASP.NET's per-process partitions onto the sharedIFixedWindowCounter(#543, Redis-backed with in-process fallback + alarm), so N replicas enforce one combined per-IP budget instead of N.Design
RateLimiter+ a publicIRateLimiterPolicy<string>live inCluckwork.Infrastructure(same assembly as the internalIFixedWindowCounterport), consumed by Api viaAddPolicy. The port stays internal — noInternalsVisibleTo Api, no port made public.RateLimitKeymoved Api → Infrastructure so the policy shares the one canonical IPv6-/64 derivation (no duplicated security control).{policyName}:{ipKey}so login/refresh/client-errors stay independent budgets (the Security: rate limiting on anonymous auth endpoints (login/refresh) #143 NAT-starvation guard), while login/step-up/change-password still share one (Production log redaction + retention + cost + log sink #273).SharedStateRedisUnavailablealarm (free, via the resilient decorator — not re-implemented).Tests
MultiInstanceRateLimitTests— two realdotnet Cluckwork.Api.dllprocesses + one real Redis Testcontainer prove one combined per-IP budget (split N across both instances, the N+1th 429s).DistributedRateLimiterWiringTests— in-process (toggleable-primary stub): fallback+alarm, recovery-without-restart, pre-account-lookup limiting, and no attacker-supplied dimension in key or log.Verified locally
Build 0/0; Domain 325 + Application 145 + Api.IntegrationTests 1256 all green. Mutation: forcing the limiter to bypass the shared counter turns the two-process combined-budget test RED (restored → green); dropping the policy prefix turns
Refresh_has_its_own_bucket_independent_of_loginRED.Scope notes
AddSingleton/AddHostedService): Security: rate limiting on anonymous auth endpoints (login/refresh) #143 is no longer an in-process scaling blocker; Bound production report workload, query size, and concurrency #311 remains (→ Distributed per-account report concurrency cap with a local ceiling fallback #545). This is not permission to scale.web/untouched. No user-visible behavior change → no GLOSSARY/Help update.