Repository navigation
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
Abdel39
added a commit
to Abdel39/airflow
that referenced
this pull request
Oct 16, 2026
Abdel39
marked this pull request as ready for review
October 16, 2026 14:20
Abdel39
requested review from
XD-DENG,
amoghrajesh,
ashb and
potiuk
as code owners
October 16, 2026 14:20
hussein-awala
left a comment
hussein-awala
left a comment
Member
There was a problem hiding this comment.
After recovery, the query excludes the Dag. The method sends no further value for that Dag. By default, StatsD keeps the last gauge value. Thus, an alert can remain active after recovery.
Please send zero for each previously reported Dag absent from the next query result. Use the existing reset pattern in _emit_ti_metrics.
Add a regression test with these steps:
- Collect the metric for a blocked Dag.
- Clear
exceeds_max_non_backfill. - Collect the metric again.
- Check for a zero value in the second collection.
hussein-awala
added
the
ready for maintainer review
…ive_runs
When a time-scheduled Dag is at max_active_runs, dags_needing_dagruns
skips it, so its next run is never created and nothing is queued. No
metric shows that the Dag is falling behind its schedule.
Emit dag.schedule_overdue_seconds{dag_id} on the dagrun_metrics_interval
timer, only for Dags in that state, so the number of series is bounded
by the number of blocked Dags.
Generated-by: Claude Opus 5.5 following the guidelines
StatsD keeps a gauge's last value, so a Dag that left the blocked state kept reporting its last lateness. Send zero once for Dags reported on the previous run but absent now, following the _emit_ti_metrics pattern.
dag.schedule_overdue_seconds and the per-dag_id scheduler.dagruns.running and scheduler.dagruns.queued gauges are emitted from scheduler queries with a hand-built dag_id tag, so they never go through DagRun.stats_tags and miss the Dag tags that [metrics] dag_tags_in_metrics adds elsewhere. Look the tags up in one DagTag query per emission and merge them the same way DagRun.stats_tags does. The zero reset for recovered Dags reuses the tags it emitted, so it lands on the same series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Abdel39
force-pushed
the
aboulfoul/schedule-overdue-seconds-metric
branch
from
November 6, 2026 16:05
ff83f48 to
e9e2877
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related: #73686
When a time-scheduled Dag is at
max_active_runs,DagModel.dags_needing_dagrunsfilters it out (exceeds_max_non_backfill == false), so its next run is never created. Nothing is queued,scheduler.dagruns.queued(#70013) stays at zero, and nothing tells you the Dag is falling behind its schedule until someone notices missing intervals.This adds a
dag.schedule_overdue_secondsgauge, tagged bydag_id: seconds since the Dag'snext_dagrun_create_after, emitted only while the Dag is due and held back bymax_active_runs.[scheduler] dagrun_metrics_intervaltimer, as a separate read-only query. It does not touchdags_needing_dagruns, which is row-locked andLIMITed.next_dagrun_create_afteralone does not make them due.We've run the same metric as a scheduler patch on our Airflow staging deployments. It was the signal teams needed to decide when to raise
max_active_runsfor scheduled Dags, where queued-run counts showed nothing.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 5.5 following the guidelines