Skip to content

feat: Add exponential backoff and log deduplication to Spotlight #5025

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattico wants to merge 4 commits into
base: main
Choose a base branch
from
+369 −31
Open

feat: Add exponential backoff and log deduplication to Spotlight #5025

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
950882d
feat: Add exponential backoff and log deduplication to Spotlight tran…
mattico Mar 13, 2026
fd08e9c
feat: introduce helper to make SpotlightHttpTransport backoff thread-…
mattico Mar 13, 2026
854aff4
Update CHANGELOG.md
mattico Mar 13, 2026
2df3b7b
chore: simplify `ExponentialBackoff` retry delay logic
mattico Mar 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- The SDK now logs a `Warning` instead of an `Error` when being ratelimited ([#4927](https://github.com/getsentry/sentry-dotnet/pull/4927))
- Symbolication now works correctly with Android workloads 10.0.102 and later ([#4998](https://github.com/getsentry/sentry-dotnet/pull/4998))
- `libmonosgen` and `libxamarin` frames no longer show as in-app ([#4960](https://github.com/getsentry/sentry-dotnet/pull/4960))
- The SDK now logs only the first consecutive error when sending envelopes to Spotlight, and uses exponential backoff before subsequent attempts ([#5025](https://github.com/getsentry/sentry-dotnet/pull/5025))

## 6.2.0-alpha.0

Expand Down
80 changes: 69 additions & 11 deletions src/Sentry/Http/SpotlightHttpTransport.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ internal class SpotlightHttpTransport : HttpTransport
private readonly HttpClient _httpClient;
private readonly Uri _spotlightUrl;
private readonly ISystemClock _clock;
private readonly ExponentialBackoff _backoff;

public SpotlightHttpTransport(ITransport inner, SentryOptions options, HttpClient httpClient, Uri spotlightUrl, ISystemClock clock)
: base(options, httpClient)
Expand All @@ -21,6 +22,7 @@ public SpotlightHttpTransport(ITransport inner, SentryOptions options, HttpClien
_spotlightUrl = spotlightUrl;
_inner = inner;
_clock = clock;
_backoff = new ExponentialBackoff(clock);
}

protected internal override HttpRequestMessage CreateRequest(Envelope envelope)
Expand All @@ -38,23 +40,79 @@ public override async Task SendEnvelopeAsync(Envelope envelope, CancellationToke
{
var sentryTask = _inner.SendEnvelopeAsync(envelope, cancellationToken);

try
if (_backoff.ShouldAttempt())
{
// Send to spotlight
using var processedEnvelope = ProcessEnvelope(envelope);
if (processedEnvelope.Items.Count > 0)
try
{
using var request = CreateRequest(processedEnvelope);
using var response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false);
await HandleResponseAsync(response, processedEnvelope, cancellationToken).ConfigureAwait(false);
// Send to spotlight
using var processedEnvelope = ProcessEnvelope(envelope);
if (processedEnvelope.Items.Count > 0)
{
using var request = CreateRequest(processedEnvelope);
using var response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(false);
await HandleResponseAsync(response, processedEnvelope, cancellationToken).ConfigureAwait(false);

_backoff.RecordSuccess();
Copy link
Copy Markdown

@sentry sentry bot Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The backoff mechanism in SpotlightHttpTransport is incorrectly reset on HTTP error responses (4xx/5xx) because _backoff.RecordSuccess() is called regardless of the response status.
Severity: HIGH

Suggested Fix

The HandleResponseAsync method should return a status indicating success. SpotlightHttpTransport should then check this status. Call _backoff.RecordSuccess() only if the response was successful (e.g., 200 OK). Otherwise, call _backoff.RecordFailure() to correctly trigger the exponential backoff logic for HTTP errors.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: src/Sentry/Http/SpotlightHttpTransport.cs#L55

Potential issue: In `SpotlightHttpTransport.SendEnvelopeAsync`, the
`_backoff.RecordSuccess()` method is called after `HandleResponseAsync`. However,
`HandleResponseAsync` does not throw an exception for non-200 HTTP status codes. If the
Spotlight server returns an HTTP error (e.g., 500, 503), the backoff state is
incorrectly reset, and the retry delay is not increased. This defeats the exponential
backoff mechanism for HTTP-level failures, causing the SDK to spam a struggling server
instead of waiting. The intended error logging for this failure path is also skipped.

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Author

@mattico mattico Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To my knowledge the python SDK does not consider an HTTP error response as a failure for backoff purposes (urllib3 doesn't throw for those). And the SDK docs don't either (only catch ConnectionError).

I haven't thought deeply about it otherwise.

}
}
catch (Exception e)
{
int failureCount = _backoff.RecordFailure();
if (failureCount == 1)
{
_options.LogError(e, "Failed sending envelope to Spotlight.");
}
}
}
catch (Exception e)
{
_options.LogError(e, "Failed sending envelope to Spotlight.");
}

// await the Sentry request before returning
await sentryTask.ConfigureAwait(false);
}

private class ExponentialBackoff
{
private static readonly TimeSpan InitialRetryDelay = TimeSpan.FromSeconds(1);
private static readonly TimeSpan MaxRetryDelay = TimeSpan.FromSeconds(60);

private readonly ISystemClock _clock;

private readonly Lock _lock = new();
private TimeSpan _retryDelay = InitialRetryDelay;
private DateTimeOffset _retryAfter = DateTimeOffset.MinValue;
private int _failureCount;

public ExponentialBackoff(ISystemClock clock)
{
_clock = clock;
}

public bool ShouldAttempt()
{
lock (_lock)
{
return _clock.GetUtcNow() >= _retryAfter;
}
}

public int RecordFailure()
{
lock (_lock)
{
_failureCount++;
_retryAfter = _clock.GetUtcNow() + _retryDelay;
_retryDelay = TimeSpan.FromTicks(Math.Min(_retryDelay.Ticks * 2, MaxRetryDelay.Ticks));
return _failureCount;
}
}

public void RecordSuccess()
{
lock (_lock)
{
_failureCount = 0;
_retryDelay = InitialRetryDelay;
_retryAfter = DateTimeOffset.MinValue;
}
}
}
}
Loading
Loading