Skip to content

Add a dag_run index on (dag_id, run_after) - #74481

Merged
ephraimbuddy merged 2 commits into
apache:mainfrom
yuseok89:add-dag-run-dag-id-run-after-index
Oct 27, 2026
Merged

ephraimbuddy merged 2 commits into
apache:mainfrom
yuseok89:add-dag-run-dag-id-run-after-index

Conversation

@yuseok89

@yuseok89 yuseok89 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

related: #74482

This adds idx_dag_run_dag_id_run_after on dag_run (dag_id, run_after), step 1 of #74482.

GET /ui/dags/{dag_id}/latest_run picks a Dag's latest run with WHERE dag_id = :dag ORDER BY run_after DESC LIMIT 1, and the Dag page polls it on every auto-refresh. dag_run has indexes on dag_id and on run_after, but none that keeps the runs of one Dag in run_after order. So the planner either reads every run of the Dag and sorts them, or scans idx_dag_run_run_after backward and filters on dag_id. With default statistics it often picks the backward scan even for Dags whose runs are old, and then reads most of the table to return one row.

Query plans

Postgres 15, a skewed synthetic dag_run with about 3.2M rows (1,400 Dags with recent runs, 400 Dags whose last run was 60 days ago, 200 Dags with one old run), the existing DagRun indexes and default statistics. The query is the one get_latest_run_info runs.

Dag Without the index With the index
One old run Backward scan of idx_dag_run_run_after, about 3.2M rows filtered out Backward scan of the new index, 1 row read
Last run 60 days ago Same scan, about 2.8M rows filtered out 1 row read
Recent runs Same scan, about 1.3K rows filtered out 1 row read

Without the index, the rows read for a Dag grow with the newer runs of every other Dag, so the cost grows with the table. With it, the lookup reads one row for any Dag. On my machine that was 640 to 760 ms against under 0.1 ms.

On MySQL 8.0 and SQLite, with the same kind of skewed data, the query also switches to the new index and stops sorting the Dag's runs. On MySQL a Dag whose last run was 60 days ago went from about 1 s to under 1 ms. SQLite already picked a dag_id index there, so the gain is small.

Index size

The new index has the same shape as the existing dag_run_dag_id_logical_date_key (dag_id plus a timestamp), so it takes about the same space.

Index on the same table Columns Size
idx_dag_run_dag_id_run_after (new) dag_id, run_after 107 MB
dag_run_dag_id_logical_date_key dag_id, logical_date 107 MB
dag_run_dag_id_run_id_key dag_id, run_id 184 MB

It took a few seconds to build on my machine.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Claude Code (Opus 5.5)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@developer-rpai developer-rpai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the latest_run query — the composite index is the right shape for WHERE dag_id = ... ORDER BY run_after DESC LIMIT 1.

A few things:

  1. migrations-ref.rst marks this revision as (head). If another 3.4.0 migration lands before this merges, the chain breaks — worth a rebase on merge day.
  2. The (dag_id, run_after) leftmost prefix covers plain dag_id lookups, so idx_dag_run_dag_id may now be redundant. Every index adds write cost to each dag-run insert, which matters for high-churn schedulers — an EXPLAIN of the latest_run query across Postgres/MySQL/SQLite would strengthen this.
  3. unique=False is correct (manual/backfill runs can share run_after), but a one-line comment in the migration would stop a future reader from "optimizing" it to unique.

@yuseok89
yuseok89 force-pushed the add-dag-run-dag-id-run-after-index branch from 7ee2f80 to c1568da Compare October 11, 2026 23:04
@yuseok89

Copy link
Copy Markdown
Contributor Author

Thanks for the review!

  1. Rebased onto the latest main. The migration now sits on top of the current head.
  2. dag_id was already the leftmost column of dag_run_dag_id_run_id_key and dag_run_dag_id_logical_date_key before this PR, so the overlap with idx_dag_run_dag_id isn't new. Dropping it would be a separate change, so I'd keep this PR to adding the index. I added the MySQL and SQLite results under the query plans in the description.
  3. unique=False is just the default the migration generator writes, so I'd leave it as is.

@yuseok89

Copy link
Copy Markdown
Contributor Author

Rebased the migration onto the new 3.5.0 head after the 3.4 branch was cut. It is now 0148_3_5_0, on top of c71065a1b14f.

@yuseok89
yuseok89 force-pushed the add-dag-run-dag-id-run-after-index branch from e020e0a to c82a0d9 Compare October 18, 2026 04:39
@yuseok89
yuseok89 force-pushed the add-dag-run-dag-id-run-after-index branch from c82a0d9 to a9ffd9a Compare October 22, 2026 14:23
@yuseok89

Copy link
Copy Markdown
Contributor Author

@ephraimbuddy, this is rebased onto the latest main and CI is green. Since each new migration on main needs another rebase here, would it be possible to merge this when you have a moment?

@ephraimbuddy

Copy link
Copy Markdown
Contributor

Please rebase again

@yuseok89

yuseok89 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks, I updated the branch with the latest main, and CI is green now.

@ephraimbuddy ephraimbuddy added this to the Airflow 3.5.0 milestone Oct 27, 2026
@ephraimbuddy
ephraimbuddy merged commit 968cca0 into apache:main Oct 27, 2026
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants