[AutomationDev85]

Overview

The scheduler currently only exports scheduler.dagruns.running as a single, untagged count — there's no way to see how many Dag runs are running or queued for a specific Dag. This makes it hard to spot Dags that are backlogged in the QUEUED state (e.g. stuck behind max_active_runs limits or pool exhaustion) or to compare running-vs-queued load per Dag in dashboards/alerts.

This change adds per-dag_id visibility by emitting both scheduler.dagruns.running and a new scheduler.dagruns.queued metric, each tagged with dag_id, so operators can track queue backlog and running by each Dag.

We look forward to your feedback.

Details of change:

• Rename _emit_running_dags_metric function to _emit_dag_runs_metric;
• Emit scheduler.dagruns.running / scheduler.dagruns.queued gauges tagged by dag_id

---

Was generative AI tooling used to co-author this PR?

• Yes -- Claude Sonnet 5

Generated-by: Claude Sonnet 5 following the guidelines

[potiuk]

Thanks — and you've correctly spotted that the old description was wrong: scheduler.dagruns.running claimed to count "DAGs whose latest DagRun is RUNNING" while the code counted DagRuns outright. Fixing that wording is worth having on its own, and adding a queued counterpart is a reasonable gap to fill.

But the change from a single aggregate gauge to one gauge per dag_id is a much bigger step than the PR presents it as, and I'd like it reconsidered before this lands. Two blocking concerns inline: metric cardinality, and the fact that this silently breaks every existing dashboard built on the old metric.

To be clear about the shape of my objection — I'm not saying per-dag visibility is wrong to want, only that turning it on unconditionally for every deployment is a decision that needs to be deliberate and documented rather than a side effect of adding a queued metric.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[potiuk]

Both of my blocking concerns are resolved. Thanks for taking the cardinality point seriously rather than arguing it — the config description is genuinely good, spelling out "one metric series per Dag with an active DagRun on every interval" so an operator can size the cost before enabling it.

On the legacy_name question I went in expecting the breakage to have simply moved into the default path, and it has not. Recording why, because it is subtle enough that the next reader may have the same worry:

• legacy_name is an additional emission, not the primary one, and legacy_name: "-" means "no separate legacy spelling" — so on main this metric already emits nothing on the legacy path.
• With the flag off the metric is emitted with tags=None, and without your early return required_vars=["dag_id"] against an empty provided_vars would raise ValueError. The stats.py change is not papering over a break; it is required to keep the default path working at all.
• Primary emission of scheduler.dagruns.running is unchanged either way, so no existing dashboard loses its series by default.

Two things left, both small.

Newsfragment

My original ask was .significant.rst, on the assumption this was breaking. That is no longer the right severity — but the PR is still user-facing: a new config option, a new scheduler.dagruns.queued metric, and a corrected description for a published one. airflow-core/newsfragments/70013.feature.rst covers it.

A test for the stats.py change

The change lives in shared/observability, which has its own suite — shared/observability/tests/observability/metrics/test_stats.py already covers legacy-name resolution. The tests added here are all in test_scheduler_job.py, so the new tags is None branch is untested where it actually lives. A case asserting tags=None returns no legacy name, and that tags={} still raises, would pin both halves of the distinction your comment draws.

One observation, not a request

That branch returns "no legacy metric" whenever tags is None for any metric with name_variables, where it previously raised. A caller who simply forgets to pass tags now gets silence rather than an error, across the whole registry rather than just these two metrics. The {}-still-raises split is a reasonable line to draw and the comment documents it — just worth having drawn it deliberately.

Also worth a re-run: the MySQL Core Serialization check is showing as CANCELLED rather than passed.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[AutomationDev85]

@potiuk Thanks for your hints! Updated the branch.

[potiuk]

Approved. Removed backporting though.