[OTEL Reinstrumentation] [RelativeLoadBalancerStrategy_Sensor, DegraderLoadBalancerStrategyV3_Sensor]#1174
Open
aadityaraj7769 wants to merge 1 commit into
Conversation
…duce HostStatus enum for improved health status tracking. Update Degrader and Relative load balancer strategy metrics providers to utilize HostStatus for reporting unhealthy and quarantined hosts. Enhance tests to validate new metrics behavior and ensure proper listener registration for tracker clients. Maintain backward compatibility while improving telemetry accuracy. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds OpenTelemetry (OTel) metrics instrumentation to two D2 load balancer strategies: RelativeLoadBalancerStrategy and DegraderLoadBalancerStrategyV3.
Changes
No-op Implementation: Added NoOpRelativeLoadBalancerStrategyOtelMetricsProvider and NoOpDegraderLoadBalancerStrategyV3OtelMetricsProvider as default implementations when metrics are disabled
Integration:
Backward Compatibility
About RelativeLoadBalancerStrategySensor
The RelativeLoadBalancerStrategy Sensor tracks metrics for D2's relative load balancer, which dynamically adjusts host health scores (0.0–1.0) based on per-host call statistics relative to the overall cluster. It monitors host latency distributions, cluster health composition, and hash ring sizing. This sensor enables server-side observability for how traffic is being distributed across hosts in a service cluster.
New OTel Metrics
Metric Naming Pattern: `D2.RelativeLb.<Metric>
Dimensions
D2.Service.NameD2.Schemehttp,https)D2.Host.StatusUnhealthyorQuarantineDegradedHostsCountonlyExponentialHistogram
D2.RelativeLb.HostLatency(ms) — Records each host's average latency per call. OTel automatically computes p50, p90, p99, average, min, max, and standard deviation from the distribution.Gauges
D2.RelativeLb.AllPartitionHostsCount({host}) — Total number of hosts across all partitions regardless of health status; the full host population the load balancer is aware ofD2.RelativeLb.DegradedHostsCount({host}) — Number of hosts in a degraded state, grouped byD2.Host.Status:D2.Host.Status = Unhealthy;— hosts whose health score has been reduced due to high latency or error rateD2.Host.Status = Quarantine;— hosts currently in quarantine pending health check recoveryD2.RelativeLb.PointsInHashRing({point}) — Total number of points in the consistent hash ring, reflecting the effective traffic weight distribution across hostsAbout DegraderLoadBalancerStrategyV3Sensor
The DegraderLoadBalancerStrategyV3 Sensor tracks metrics for D2's degrader load balancer, which uses degradation-based health tracking to adaptively route traffic. Each host is placed on a consistent hash ring with points proportional to its health — healthy hosts receive more traffic while degraded hosts are shed. When the entire cluster is unhealthy, the strategy applies a cluster-wide override drop rate that proactively drops a fraction of requests before they reach any host, protecting against cascading failures. This sensor enables server-side observability for the degrader's core health signals: host latency distribution, cluster-level drop rates, and hash ring capacity.
New OTel Metrics
Metric Naming Pattern: `D2.DegraderLb.<Metric>
Dimensions
ExponentialHistogram
D2.DegraderLb.HostLatency(ms) — Records each host's per-call latency. OTel automatically computes standard deviation, percentiles (p50, p90, p99), average, min, and max from the distribution. These derived values map to the degrader's key health signals: LatencyStandardDeviation, MaxLatencyRelativeFactor, and Pct99LatencyRelativeFactor.Gauges
D2.DegraderLb.OverrideClusterDropRate({ratio}) — Current cluster-level override drop rate in range [0.0, 1.0]; indicates how aggressively the cluster is shedding traffic due to widespread degradationD2.DegraderLb.PointsInHashRing({point}) — Total number of points across all hosts in the consistent hash ring; a drop in total points signals that hosts are being degraded or quarantined, reducing overall cluster capacity