HTTP Health check spam in telemetry after adding health checks to template

[davidfowl]

After this #8444

Feels like this should be filtered out, related to #4198

cc @DamianEdwards

[DamianEdwards]

Hmm how would we approach a filter here? These are standard incoming requests traces right? Is there a simple mechanism to filter traces/activities based on properties?

[afscrome]

What about executing all health checks with a single TraceId - that way they'd still show up on the dashboard, but they'd all be under one trace so they'd only spam if you opened the healthcheck trace. (In which case you probably do want to see the individual items).

[DamianEdwards]

What about executing all health checks with a single TraceId

These are different resources though. This isn't a trace for the health checks processing by the app host, it's a standard incoming request trace to the health checks endpoint on each project that has had the WithHttpsHealthCheck("/health") call added. If we want to filter these out I think we'd need to configure OTel in Service Defaults to filter out the traces for these requests.

[DamianEdwards]

Had a quick look and I think this would require adding a custom processor to the tracing setup in service defaults? There doesn't appear to be a built-in processor for simple filtering and the methods to add ASP.NET Core instrumentation don't appear to support any kind of filtering either.

/Cc @samsp-msft

[DamianEdwards]

OK, with fresh eyes this morning I figured it out:

.WithTracing(tracing =>
{
    tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            options.Filter = context =>
            {
                // Exclude health check requests from tracing
                return !context.Request.Path.StartsWithSegments("/health")
                       && !context.Request.Path.StartsWithSegments("/alive");
            };
        })
        .AddGrpcClientInstrumentation()
        .AddHttpClientInstrumentation();
});

I'll send a PR to update the templates.