Repository navigation
Description
Description
Since scheduler dag resolution became version-aware, resolving a dag run's dag goes through DagVersion.get_latest_version — an uncached point SELECT issued once per call. Two scheduler code paths still resolve per item, so their query count scales with the number of items processed per tick:
-
Critical section, per-task-instance resolution —
SchedulerJobRunner._executable_task_instances_to_queuedcallsscheduler_dag_bag.get_dag_for_run(...)per candidate task instance when the dag has task-concurrency limits (thehas_task_concurrency_limitsbranch inairflow-core/src/airflow/jobs/scheduler_job_runner.py). Each call issues oneDagVersion.get_latest_versionSELECT; withmax_tis_per_query(default 512) candidates, deployments usingmax_active_tis_per_dag/max_active_tis_per_dagruncan pay hundreds of point queries per tick inside the scheduler's most latency-sensitive section. -
DagRun.version_numberlazy loads in dag-run state listeners —on_dag_run_runningimplementations that readdag_run.version_number(the example-plugin listener does; the OpenLineage provider listener shares the hook) trigger per-run lazy loads of thedag_versionsrelationship — roughly 1.7 extra queries per started run when measured. Eager-loading the relationship inDagRun.get_queued_dag_runs_to_set_running, or batching at the listener call sites, would remove this.
Both are measurable with tests_common.test_utils.asserts.CountQueries around the respective loops: query counts grow linearly with the number of items instead of staying constant per loop.
Use case/motivation
The scheduler runs these loops on every tick, so redundant per-item version lookups add database round-trips and latency directly on the scheduling hot path, and the cost grows with the number of task instances and dag runs a deployment examines per tick. The goal is for version resolution within a single scheduler loop to cost a constant number of queries regardless of how many items the loop processes — resolved once per batch (or at minimum once per dag) — mirroring the batching the main scheduling loops already use.
Acceptance criteria:
- Version lookups in the critical section do not scale with the number of candidate task instances.
- Starting queued runs does not trigger per-run
dag_versionslazy loads from core-shipped listeners. - Query-count regression coverage using
CountQueriesasserting per-loop constants.
Related issues
- Save scheduler execution time by caching dags #30704 added per-loop caching of dag resolution in the scheduler to avoid repeated lookups.
- Ensure scheduler uses best available serdag for dag run #49097 made scheduler dag resolution version-aware (routing it through
DagVersion.get_latest_version).
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Moemenmohamed24 commentedon Jul 11, 2026
Hi , I would like to work on this feature, would you grant me assign?