Memory leak/issues with UnsampledTransactions in 5.9.0

[jhoeb]

Package

• Sentry
• Sentry.AspNetCore

.NET Flavor

.NET

.NET Version

8.0.21

OS

Linux

OS Version

3.21

Development Environment

Visual Studio v17.x

SDK Version

5.9.0

Self-Hosted Sentry Version

No response

Workload Versions

n/a

UseSentry or SentrySdk.Init call

We have the problem when calling:
builder.WebHost.UseSentry();

The problems are resolved when calling:

builder.WebHost.UseSentry(opts =>
{
    opts.AutoRegisterTracing = false;
});

Steps to Reproduce

Only able to reproduce in production with a service. No way of reproducing the issue in a development environment. It looks like it has to do with a very large number of incoming and outgoing http calls all triggering middleware.

Expected Result

When running on 5.8.1, the service has low memory usage. The information below is from a memory-dump dumpheap -stat running 5.8.1 in production.
When disabling tracing on 5.9.0 AutoRegisterTracing = false, the memory stays low as wel.

7fe95ea0cb98  19,490  2,026,960 System.Reflection.RuntimeMethodInfo
7fe9624690a8       5  2,097,304 System.Threading.ThreadLocal<System.Collections.Concurrent.ConcurrentBag<Sentry.SentryPackage>+WorkStealingQueue>+LinkedSlotVolatile[]
7fe95dec9f78  18,522  2,176,464 System.Collections.Generic.Dictionary<System.String, System.String>+Entry[]
7fe95d90d4c0  34,554  2,764,320 System.Collections.Generic.Dictionary<System.String, System.String>
7fe959dfbf68  51,997  2,782,976 System.Object[]
7fe961bd1f98      16  2,866,176 System.Collections.Generic.Dictionary<System.Int32, System.Boolean>+Entry[]
7fe967100ff0  38,562  3,084,960 System.Security.Claims.Claim
7fe961c00088  14,151  4,528,320 System.Collections.Concurrent.ConcurrentDictionary<System.String, System.Object>+VolatileNode[]
7fe959ea89e0  68,417  5,534,496 System.Int32[]
7fe95a290780  84,198  8,965,784 System.String[]
7fe95df5f5f0  61,515 13,021,087 System.Byte[]
7fe9d87db0c0  73,668 35,394,368 Free
7fe959ead7c8 489,356 55,278,876 System.String
Total 4,135,647 objects, 336,192,876 bytes

Actual Result

This is the memory-dump dumpheap -stat running 5.9.0 without AutoRegisterTracing = false. The memory shoots up and the service crashes with an OOM exception within one hour.

7ff875310780    93,028  12,813,328 System.String[]
7ff882736fc8   404,955  16,198,200 System.Collections.Concurrent.ConcurrentBag<Sentry.ISpan>
7ff882d52a20   406,813  16,272,520 Sentry.Internal.UnsampledSpan
7ff882737378   408,505  16,340,200 System.Threading.ThreadLocal<System.Collections.Concurrent.ConcurrentBag<Sentry.ISpan>+WorkStealingQueue>
7ff879136520    64,133  19,462,435 System.Byte[]
7ff8f3850c10    92,778  21,791,320 Free
7ff882d52ca0         6  25,165,968 System.Threading.ThreadLocal<System.Collections.Concurrent.ConcurrentBag<Sentry.ISpan>+WorkStealingQueue>+LinkedSlotVolatile[]
7ff8827372b8   405,864  25,975,296 System.Collections.Concurrent.ConcurrentBag<Sentry.ISpan>+WorkStealingQueue
7ff87531e228   454,845  36,387,600 System.Collections.Generic.Dictionary<System.String, System.String>
7ff874f289e0   478,717  36,554,152 System.Int32[]
7ff8827374a8   814,101  39,076,848 System.Threading.ThreadLocal<System.Collections.Concurrent.ConcurrentBag<Sentry.ISpan>+WorkStealingQueue>+LinkedSlot
7ff882736a18   404,955  48,594,600 Sentry.Internal.UnsampledTransaction
7ff882380818   404,957  48,594,840 Sentry.TransactionContext
7ff882d52ec0   405,864 113,641,920 Sentry.ISpan[]
7ff87531e9b0   430,084 119,115,840 System.Collections.Generic.Dictionary<System.String, System.String>+Entry[]
7ff874f2d7c8 2,385,183 172,165,504 System.String
Total 14,108,911 objects, 1,159,024,504 bytes

[linear]

NET-517

[Flash0ver]

Thanks @jhoeb for the report.
In particular, thank you very much for the memory stats and describing the behavior per scenario.

I have the feeling you're quite onto it already.
With your info, we were able to craft a quick reproduction:

SentrySdk.Init(options =>
{
    // ..
    options.EnableBackpressureHandling = false; // to simulate v5.9.0 behavior
    options.TracesSampleRate = 1.0; // memory stable (Sentry.TransactionTracer - Sentry.SpanTracer)
    options.TracesSampleRate = 0.0; // memory ever increasing (Sentry.Internal.UnsampledTransaction - Sentry.Internal.UnsampledSpan)
    options.Transport = NullTransport.Instance;
});

while (true)
{
    using var transaction = SentrySdk.StartTransaction("name", "operation");
    using var span = transaction.StartChild("child");
    Console.WriteLine($"{transaction} - {span}");
}

sealed class NullTransport : ITransport
{
    public static ITransport Instance { get; } = new NullTransport();

    public Task SendEnvelopeAsync(Envelope envelope, CancellationToken cancellationToken = default)
    {
        return Task.CompletedTask;
    }
}

The root cause could be that the Sentry.Internal.UnsampledTransaction's Sentry.Internal.UnsampledSpans never get Cleared, so instances of these types forever referencing each other, never being subject to Garbage Collection.

sentry-dotnet/src/Sentry/Internal/UnsampledTransaction.cs

Line 15 in 9e2f5f2

private readonly ConcurrentBag<ISpan> _spans = [];

I'm afraid I need to head off ... but I'll work on a fix once back at the machine.
But this should be a quick fix ... thanks to the insights you provided.

[jamescrosswell]

We should probably backport this to version 5, once we've merged version 6 into the main branch... I'd rather not do it before then, since there's a merge conflict when trying to cherry pick this, that might cause complications if we do it sooner.