[Rate Limiting] Design & Implement Global Multi-Region Rate Limiting

[Olshansk]

Warning

See the comments in the thread for next steps: #11 (comment)

Objective

Enable multi-region global rate limiting for portal.

How

Using the Envoy Gateway gRPC Quota IDL --> communicating to PEAS --> backed by redis and/or portalDB.

WIP

What is the simplest way to:

• Have PEAS communicate with Envoy Global Rate Limiters
• Reduce or remove logic on portal workers
• Remove reliance on portalDB
• But also do this:

Origin Document

• https://www.envoyproxy.io/docs/envoy/latest/configuration/other_features/rate_limit#rate-limit-quota-service
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/rate_limit_quota/v3/rlqs.proto#envoy-v3-api-file-envoy-service-rate-limit-quota-v3-rlqs-proto
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/ratelimit/v3/rls.proto#envoy-v3-api-msg-config-ratelimit-v3-ratelimitserviceconfig
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/rate_limit_quota/v3/rate_limit_quota.proto#envoy-v3-api-field-extensions-filters-http-rate-limit-quota-v3-ratelimitquotabucketsettings-no-assignment-behavior
• https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/global_rate_limiting

[Olshansk]

@commoddity Don't read this yet, but just #FYI that this is the work I want to design & scope for next week.

[Olshansk]

tl;dr Raw & unedited notes from adhoc call with @fredteumer @commoddity @okdas

Rough Diagram

Options

1. Extending PEAS

• Built around gRPC interface
• Auth decisions

2. Envoy Patch Policy

• Requires more research

3. [Parallel] Wait for EnvoyGateway RLQS

4. [Avoid] Bespoke

5. [Doesn't work] One redis clusters all regions (latency)

Definitions

Rate Limiting:

• Just-in-time
• DDoss protection

Quota Limiting:

• Monthly limits
• Eventually consistent

Goals

1. Eventually consistent monthly rate limits
2. As simle as possible

Next Steps

Pascal: owner of quota limiting ($$$, month, peas, grove, etc)
Dima: owner of throuput limiting (ddoss, rps, dns, infra, etc)

---

[Bear Pascal] Solution:

1. PEAS periodically queries data warehouse (e.g. every 10 minutes)
2. PEAS maintains in-memory map for quota limiting (no new infra/caching)
3. PEAS "monthly quota limit" returns an error through check to in-mem map
4. QED

Technical implementation:

• Warm-up on PEAS launch: query data warehouse
• Race-condition on PEAS launch: allow requests

[Dima] Side work:

• Redis from global -> local rate limiting (ddoss)
• Ddos protection
• Throughput RPS

Tradeoffs:

• Eventually consistent monthly quota limiting
• Cost to run queries on data warehouse
• Re-evaluate tradeoffs after it's live

[Olshansk]

Owners & responsibilities:

@okdas: Throughput rate limiting (RPS, ddoss protection, network driven)
@commoddity: Quota rate limiting (Monthly quotas, eventually consistent, business logic driven)

Deliverables & Next Steps:

1. @okdas Convert redis based global rate limiting to local rate limiting in our redis clusters
2. @commoddity to modify PEAS such that:
   • It keeps an in-memory cache of user quotas
   • Periodically query data warehouse for aggregate quota counts
   • Update "request authorization" to check against monthly quotas

Tradeoffs & Decisions:

1. Quota-based rate limiting will be eventually consistent (may be out of date by 5-15 minutes)
2. Leverage the existing External Authorization service instead of addint the complexity of adding External Processing
3. Cost of data warehouse queries will be re-evaluated after its live
4. Resource cost of an in-memory "cache solution" will be re-evaluated after its live
5. Not using Envoy Patch because it's too complex
6. Not using Rate Limit Quota Service because it's not supporting by Envoy Gateway as discussed here: Async rate-limiting envoyproxy/gateway#5457