Alert - Be able to trigger 1 alert if an issue hits X threshold for the first time

[Angelodaniel]

Problem Statement

Here's the general use case; maybe there's another way to do this. Use case is... I want 1 alert (1 slack thread in an alerts channel) per issue. I want it created when the issue first becomes alert-worthy. We'll expect each one of those slack threads to be reviewed and actioned.

I don't want to just use "New issue" because that's too noisy; 1 event is not alert-worthy. Ideally it'd be like "the first time the issue has 10 events within 1 day" or "the first time the issue affects 10 users within 1 day".

The "action interval" feature comes close to what I want, but it's limited to 30 days, which would mean that every issue might have to get re-triaged every 30 days, and that'll be a lot of noise. We really care about what's new.

Solution Brainstorm

No response

Product Area

Alerts

[linear-code]

ENG-7190

[getsantry]

Auto-routing to @getsentry/product-owners-alerts for triage ⏲️

[sentaur-athena]

@Angelodaniel how is this different from a metric alert on event count with interval of 1 day?
On another note, we can't have issues that last long because of data retention, so at some point there will be a new issue if not taken care of. What's the use case you're trying to slove?

[Angelodaniel]

@sentaur-athena the goal is to have the alert fire once "ever" if the threshold is hit.
Metric alert with interval would just go up and down and won't specify the issue it triggered on so it's not actionable I suppose.
Issue don't have data retention, the error's underneath it does but the issues itself live fort as long as there is events getting in.
So it would be possible to have long lived issues that don't hit the threshold.

Customer:

how is this different from a metric alert on event count with interval of 1 day?

That could alert every day, I just want 1 alert the first time.

we can't have issues that last long because of data retention, so at some point there will be a new issue if not taken care of

We could live with that limitation.

[sentaur-athena]

@Angelodaniel Actually that's not quite true anymore. Metric alerts now create issues and going to that issue you can investigate what's wrong and a list of contributing issues. It's not launched to all customers yet but you can see examples of it on sentry org.

In terms of having an alert on count of events that only fires once, I can't see a common usecase for it. For example, metric alert says we hit the number of events, alert, it's fixed today, it regresses tomorrow you would want another alert, why wouldn't you?