[safaehar]

Emit three new metrics from `SchedulerJobRunner` to improve visibility into scheduler health. These metrics have been running in production at Datadog via local monkey-patches and are contributed upstream so the patches can eventually be removed.

New metrics

• `scheduler.loop_exceptions{exception_class}` — counter incremented when the scheduler loop exits with an unhandled exception, tagged with the exception class. Emitted from the existing `except Exception` block in `_execute`.

• `scheduler.executor_events.batch_size` (gauge) and `scheduler.executor_events.processed` (counter) — emitted on every successful call to `process_executor_events` with the size of the event batch. `scheduler.executor_events.failed{exception_class}` is incremented when the call raises, tagged with the exception class.

• `scheduler.zombies.detected{reason}` — counter incremented when zombie task instances are detected, tagged by detection path:

  • `heartbeat_timeout`: task exceeded the heartbeat threshold

Tests

New test class `TestSchedulerObservabilityMetrics` in `airflow-core/tests/unit/jobs/test_scheduler_job.py` covers all three metrics (success and failure paths):

pytest airflow-core/tests/unit/jobs/test_scheduler_job.py::TestSchedulerObservabilityMetrics -v

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below) : Claude

---

Important

🛠️ Maintainer triage note for @safaehar · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @hussein-awala is waiting on you:

• 1 unresolved review thread(s) from @hussein-awala need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[Copilot]

Pull request overview

Adds scheduler observability metrics in SchedulerJobRunner to improve visibility into scheduler loop failures, executor event processing volume/failures, and zombie task detection, along with unit tests to validate emission.

Changes:

• Emit scheduler.loop_exceptions{exception_class} when _execute() exits due to an unhandled exception.
• Emit executor event batch metrics (scheduler.executor_events.batch_size, scheduler.executor_events.processed, scheduler.executor_events.failed{reason}) around process_executor_events().
• Emit scheduler.zombies.detected{reason} for heartbeat-timeout zombies and orphan adopt/reset failures, with new/updated unit tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
airflow-core/src/airflow/jobs/scheduler_job_runner.py	Adds new scheduler metrics for loop exceptions, executor event processing success/failure, and zombie detection.
airflow-core/tests/unit/jobs/test_scheduler_job.py	Updates existing executor-event assertions and adds a new test class covering the new metrics.

[potiuk]

@safaehar A few things need addressing before review — see our Pull Request quality criteria.

• ❌ Pre-commit / static checks. See docs.

No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[safaehar]

Thanks @potiuk, addressed the failing static checks in ab239f6:

• check-metrics-synced-with-registry: registered the five new metrics (scheduler.loop_exceptions, scheduler.executor_events.{processed,failed,batch_size}, scheduler.zombies.detected) in shared/observability/src/airflow_shared/observability/metrics/metrics_template.yaml.
• ruff format: applied (ruff 0.15.15, matching the pinned pre-commit version).

All three hooks (ruff, ruff-format, check-metrics-synced-with-registry) now pass locally.

[hussein-awala]

LGTM once the doc is updated.

Needs a second review before merging.

[potiuk]

LGTM on the metrics — correct, cheap (once per scheduler loop, no added DB round-trips), registered in the metrics template, and tested.

One ask before merge: please update the description — it still advertises scheduler.zombies.detected{reason:adopt_failure}, which was removed during review (only heartbeat_timeout emits now), so the advertised metric surface no longer matches the code.

Minor (non-blocking): the executor_events emit block is duplicated at all three return points; a small helper would keep batch_size/processed from drifting.

[hussein-awala]

All comments and suggestions have been addressed. I'll merge this now.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[hussein-awala]

Congrats @safaehar on your first commit!