Repository navigation
Add a dag_run index on (dag_id, run_after) - #74481
Merged
ephraimbuddy merged 2 commits intoOct 27, 2026
Merged
Conversation
developer-rpai
left a comment
Contributor
There was a problem hiding this comment.
Good catch on the latest_run query — the composite index is the right shape for WHERE dag_id = ... ORDER BY run_after DESC LIMIT 1.
A few things:
- migrations-ref.rst marks this revision as (head). If another 3.4.0 migration lands before this merges, the chain breaks — worth a rebase on merge day.
- The (dag_id, run_after) leftmost prefix covers plain dag_id lookups, so idx_dag_run_dag_id may now be redundant. Every index adds write cost to each dag-run insert, which matters for high-churn schedulers — an EXPLAIN of the latest_run query across Postgres/MySQL/SQLite would strengthen this.
- unique=False is correct (manual/backfill runs can share run_after), but a one-line comment in the migration would stop a future reader from "optimizing" it to unique.
yuseok89
force-pushed
the
add-dag-run-dag-id-run-after-index
branch
from
October 11, 2026 23:04
7ee2f80 to
c1568da
Compare
Contributor
Author
|
Thanks for the review!
|
ephraimbuddy
approved these changes
Oct 13, 2026
Contributor
Author
|
Rebased the migration onto the new 3.5.0 head after the 3.4 branch was cut. It is now |
yuseok89
force-pushed
the
add-dag-run-dag-id-run-after-index
branch
from
October 18, 2026 04:39
e020e0a to
c82a0d9
Compare
yuseok89
force-pushed
the
add-dag-run-dag-id-run-after-index
branch
from
October 22, 2026 14:23
c82a0d9 to
a9ffd9a
Compare
Contributor
Author
|
@ephraimbuddy, this is rebased onto the latest main and CI is green. Since each new migration on main needs another rebase here, would it be possible to merge this when you have a moment? |
ephraimbuddy
approved these changes
Oct 26, 2026
Contributor
|
Please rebase again |
Contributor
Author
|
Thanks, I updated the branch with the latest main, and CI is green now. |
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related: #74482
This adds
idx_dag_run_dag_id_run_afterondag_run (dag_id, run_after), step 1 of #74482.GET /ui/dags/{dag_id}/latest_runpicks a Dag's latest run withWHERE dag_id = :dag ORDER BY run_after DESC LIMIT 1, and the Dag page polls it on every auto-refresh.dag_runhas indexes ondag_idand onrun_after, but none that keeps the runs of one Dag inrun_afterorder. So the planner either reads every run of the Dag and sorts them, or scansidx_dag_run_run_afterbackward and filters ondag_id. With default statistics it often picks the backward scan even for Dags whose runs are old, and then reads most of the table to return one row.Query plans
Postgres 15, a skewed synthetic
dag_runwith about 3.2M rows (1,400 Dags with recent runs, 400 Dags whose last run was 60 days ago, 200 Dags with one old run), the existingDagRunindexes and default statistics. The query is the oneget_latest_run_inforuns.idx_dag_run_run_after, about 3.2M rows filtered outWithout the index, the rows read for a Dag grow with the newer runs of every other Dag, so the cost grows with the table. With it, the lookup reads one row for any Dag. On my machine that was 640 to 760 ms against under 0.1 ms.
On MySQL 8.0 and SQLite, with the same kind of skewed data, the query also switches to the new index and stops sorting the Dag's runs. On MySQL a Dag whose last run was 60 days ago went from about 1 s to under 1 ms. SQLite already picked a dag_id index there, so the gain is small.
Index size
The new index has the same shape as the existing
dag_run_dag_id_logical_date_key(dag_idplus a timestamp), so it takes about the same space.idx_dag_run_dag_id_run_after(new)dag_id,run_afterdag_run_dag_id_logical_date_keydag_id,logical_datedag_run_dag_id_run_id_keydag_id,run_idIt took a few seconds to build on my machine.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.