Proposal: ORCA-driven locality routing for zone-aware load balancing

[jukie]

Description:

Most of my orgs infra resides in on-premise datacenters with many racks, each of which is treated as an individual locality zone in the context of Envoy. For our large services we use least-request + zone-aware routing to balance traffic while preferring the local rack. This works well when incoming load is roughly uniform between racks/zones, but when it isn't the upstream cluster develops hotspots.

The cause here is that zone-aware routing decides what percentage of traffic to keep local vs remote based on healthy host counts (default) or host weights. It has no visibility into actual backend load. Least-request helps within a zone but can't redistribute across zones.

Proposed solution: load-aware locality_basis mode for zone-aware routing

Add an ORCA_UTILIZATION (open to better names) locality_basis mode to ZoneAwareLbConfig that uses ORCA load reports to drive the local-vs-remote split:

1. Aggregate per-zone average utilization from ORCA reports on each host.
2. Compare local zone utilization to the global average.
3. If the local zone is overloaded, spill excess traffic to underloaded remote zones proportional to their available headroom (1 - utilization).
4. (optional) Apply EMA dampening to smooth out ORCA data and prevent oscillation.
5. Support a utilization variance threshold knob — when all zones are within threshold of the global average, snap to local-only routing to avoid unnecessary cross-zone traffic.

This approach converges toward utilization equilibrium across zones while also preserving local preference as much as possible.

Prerequisites / related work:

This mode requires an LB policy that populates ORCA data and accepts localityLbConfig or zoneAwareLbConfig directly. I've started on this in #43623 by:

• Extracting code from client-side weighted round robin into an OrcaWeightManager class that handles ORCA report processing, per-host weight calculation (weight = qps / utilization), periodic updates, and median-weight fallback for hosts without ORCA data. This is nearly identical to the existing code and is intended to be reusable across ORCA-aware LB policies.
• Rewriting wrr_locality as a standalone ThreadAwareLoadBalancer that accepts full localityLbConfig and creates an OrcaWeightManager, rather than wrapping client-side weighted round robin.

Once the above is added I have a prototype ready for the locality_basis mode and will open a follow-up PR.

Why a new locality basis mode vs HEALTHY_HOSTS_WEIGHT?

The OrcaWeightManager extraction + localityLbConfig integration means that HEALTHY_HOSTS_WEIGHT locality basis would use ORCA-derived host weights instead for the zone split but it solves a different problem. The ORCA weight formula is weight = qps / utilization, which measures host capacity — how much throughput a host can deliver per unit of utilization. For well-behaved backends where utilization scales linearly with QPS, this simplifies to a constant regardless of current load. A host at 80% utilization and a host at 30% utilization with the same hardware get the same weight.

This means HEALTHY_HOSTS_WEIGHT is useful for capacity asymmetry (heterogeneous hardware, partially degraded hosts) but is blind to load asymmetry (uneven incoming traffic causing hotspots), which is the primary problem motivating this proposal. ORCA_UTILIZATION addresses this by routing based on available headroom (1 - utilization) directly rather than capacity-derived weights.

Alternative considered: new ORCA-based LB policy with locality selection

Rather than extending zone-aware routing, a dedicated ORCA-based policy could handle locality selection internally — potentially via a new locality_picking_policy alongside endpoint_picking_policy. This would give full control over the locality selection algorithm but wouldn't benefit from existing zone-aware routing infrastructure (residual routing, panic thresholds, etc.) and would be a larger design surface.

I believe extending zone-aware routing is the better path since it builds on proven mechanics and may be easier for users to reason about, but I'd like additional input on whether a separate locality selection abstraction has broader value.

Relevant Links:

• Open Request Cost Aggregation (ORCA) #6614
• wrr_locality: refactor as a standalone loadbalancing policy #43623

[markdroth]

It seems better to have a new policy to do ORCA-based locality selection rather than extending WrrLocality to do this.

In general, the whole reason for moving toward a model where an endpoint-picking policy like CSWRR sits underneath a locality-picking policy like WrrLocality is so that we can choose each policy independently of the other. It should be possible to change the locality-picking policy without changing the endpoint-picking policy by swapping out WrrLocality for some other policy but retaining the same child policy. And it should be possible to change the endpoint-picking policy without changing the locality-picking policy by swapping out CSWRR for some other policy but retaining WrrLocality as the parent policy.

The WrrLocality policy is specifically intended to implement locality picking based on locality weights from EDS. If what you want here is to instead do locality picking based on ORCA data, then I think it makes sense to have a different policy for that.

[jukie]

Thanks for the feedback @markdroth . I want to make sure I understand your desires before adjusting things.

My interpretation of WrrLocality based on the proto literally saying ("Weighted Round Robin Locality-Picking") and the linked ORCA issue was that it's the policy responsible for handling the locality-picking logic. It's initial implementation was a thin wrapper over CSWRR but adds respect for EDS weights (e.g. localityWeightedLbConfig), but I didn't think it was intended to stop there.

Do you envision a separate LB policy for each locality selection strategy? And if so, would there eventually be a single LB policy that supports all of them with a locality_picking_policy field? I'm trying to understand why localityLbConfig is the wrong entrypoint for this.

I can update my implementation as a new lb policy instead but a few more follow-ups:

1. Should the utilization-based locality selection logic be self-contained in the new policy or are you okay with the approach of integrating with zone aware routing as a locality_basis mode?
2. Any objections to extracting ORCA weight management code from CSWRR into a reusable OrcaWeightManager class or do you suggest a different direction for handling things in the new policy?

[markdroth]

My interpretation of WrrLocality based on the proto literally saying ("Weighted Round Robin Locality-Picking") and the linked ORCA issue was that it's the policy responsible for handling the locality-picking logic.

It is A policy responsible for locality-picking logic, not THE policy for locality-picking logic. Just as there are different LB policies for different endpoint-picking algorithms (e.g., CSWRR vs. least_request), we want there to be different LB policies for different locality-picking policies.

Historically, Envoy has bundled both locality picking and endpoint picking into a single policy. And there are some cases where that makes sense, like ring_hash. But there will also be plenty of cases where it doesn't, where we want the ability to choose the two indepdently of each other.

It's initial implementation was a thin wrapper over CSWRR but adds respect for EDS weights (e.g. localityWeightedLbConfig), but I didn't think it was intended to stop there.

It's not actually supposed to be CSWRR-specific in any way. Envoy copied this API from gRPC (which also uses the xDS API), and in gRPC the WrrLocality policy can have any endpoint-picking policy as its child policy. But when @efimki implemented WrrLocality in Envoy, he needed it only for CSWRR, and it would have taken more work to generalize it, so as a short-term hack, he made it CSWRR-specific. I'd love to see that changed so that WrrLocality can be used with any endpoint-picking policy.

Do you envision a separate LB policy for each locality selection strategy?

Yes. As I said above, just as there are multiple LB policies for various endpoint-picking policies, there should be multiple LB policies for locality-picking policies, and users should be able to choose the policy they want for each layer independently of the other.

And if so, would there eventually be a single LB policy that supports all of them with a locality_picking_policy field?

No, that is not what we want. If we have a completely separate LB policy for each locality-picking algorithm, then there's no need for a single policy with a locality_picking_policy field. We just need each locality-picking policy to have its own config field to configure the endpoint-picking policy underneath it, just like WrrLocality does.

I'm trying to understand why localityLbConfig is the wrong entrypoint for this.

There's a bunch of history here. Historically, Envoy had only a fixed list of LB policies, selected via an enum in the CDS resource. Each policy was responsible for both locality-picking and endpoint-picking in a single monolithic policy, and the strategy for being able to configure locality-picking and endpoint-picking independently was to have the locality-picking code in a common base class that all LB policies inheritted from. The CDS resource contained a CommonLbConfig message, which was used to configure things implemented by the common base class, including locality picking behavior.

In contrast, gRPC had developed a more modular approach, where LB policies are pluggable, and locality-picking policies are actually independent policies that wrap around endpoint-picking policies. When gRPC started using xDS, one of the first changes we contributed was to add a new extension point for LB policies to xDS, so that we could use xDS to configure our LB policies.

Going forward, this is the model we want Envoy to use as well, but the transition to that world has been very gradual. When Envoy first implemented support for the new LB policy extension point, it was desirable to continue to support all of its existing policies via the new extension point. And since those existing policies handle both locality-picking and endpoint-picking in the same policy, there was a need to continue to have some configuration for the locality-picking functionality implemented in the base class. As a result, we created the LocalityLbConfig message, which is essentially just the extension-friendly form of the legacy CommonLbConfig message from the CDS resource. But it's still there only to support the existing LB policies that were already structured that way; this is not intended to be the right way to configure locality-picking policies going forward.

1. Should the utilization-based locality selection logic be self-contained in the new policy or are you okay with the approach of integrating with zone aware routing as a locality_basis mode?

From an xDS API perspective, I would prefer not to use the LocalityLbConfig message. Instead, I suggest writing a new config proto that contains the knobs that you want to support for your new policy specifically.

From an Envoy implementation perspective, it seems fine to use the existing zone-aware routing implementation. I'm not suggesting that we reimplement something that already works; I just want the fact that we're using common code here to be an implementation detail of Envoy, rather than hard-coding Envoy's architectural choices into the xDS API.

3. Any objections to extracting ORCA weight management code from CSWRR into a reusable OrcaWeightManager class or do you suggest a different direction for handling things in the new policy?

I have no objection to that, but it's ultimately an Envoy implementation detail, and I can't really speak authoritatively on that -- my expertise here is primarily about the xDS API. But @efimki may have thoughts on this.

[jukie]

Thank you very much, this is super helpful!

[jukie]

I'm planning on breaking this up into 3 parts. Please tell me if this sounds reasonable:

1. Extract ORCA logic from CSWRR into reusable library #43695 - Extract OrcaWeightManager from CSWRR into a reusable common library so the new policy can use it directly.
2. Initial MVP policy: Self-contained LB policy that manages its own OrcaWeightManager instance and does its own WRR endpoint selection. The policy would read ORCA report data and aggregate utilization per-locality to calculate routing percentages. Note this would not include a child policy config.
3. Add per-locality PrioritySet views, which I think means a child LB per-locality, and expose endpoint_picking_policy with support for CSWRR, LeastRequest, etc.

[markdroth]

Steps 1 and 3 sound fine to me.

I would really prefer not to do step 2. That would introduce another new LB policy API that handles both locality picking and endpoint picking in a single monolithic policy, and we'd basically never be able to get rid of that API once we introduce it. I pushed back against this when we introduced CSWRR for the same reason.

[jukie]

Sounds good. I'll try and keep the scope as small as possible.