Repository navigation
Speed up Dags list and dashboard queries on large DagRun tables - #67721
Merged
pierrejeambrun merged 3 commits intoJun 10, 2026
Merged
Conversation
yuseok89
marked this pull request as ready for review
May 30, 2026 03:52
yuseok89
requested review from
bugraoz93,
choo121600,
ephraimbuddy,
jason810496,
pierrejeambrun,
rawwar and
shubhamraj-git
as code owners
May 30, 2026 03:52
choo121600
approved these changes
May 30, 2026
choo121600
left a comment
choo121600
left a comment
Member
There was a problem hiding this comment.
Overall looks good to me :)
potiuk
added
the
ready for maintainer review
1 task done
2 tasks done
pickfire
added a commit
to pickfire/airflow
that referenced
this pull request
Oct 7, 2026
…ables The per-Dag UNION ALL branches introduced in apache#67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-apache#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping apache#67721's exact-N-per-Dag semantics (row_number ≡ per-branch LIMIT, including ties — the pre-apache#67721 rank() returned extra rows on run_after ties) and fetching all columns inline (no join-back). Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s response byte-identical to the union (same rows, same order, 291/291 id-set match, total_entries unchanged) The window-function shape is portable (same ANSI syntax as the pre-apache#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com>
pickfire
added a commit
to pickfire/airflow
that referenced
this pull request
Oct 7, 2026
…ables The per-Dag UNION ALL branches introduced in apache#67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-apache#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping apache#67721's exact-N-per-Dag semantics (row_number ≡ per-branch LIMIT, including ties — the pre-apache#67721 rank() returned extra rows on run_after ties) and fetching all columns inline (no join-back). Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s response byte-identical to the union (same rows, same order, 291/291 id-set match, total_entries unchanged) The window-function shape is portable (same ANSI syntax as the pre-apache#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com>
pickfire
added a commit
to pickfire/airflow
that referenced
this pull request
Oct 7, 2026
…ables The per-Dag UNION ALL branches introduced in apache#67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-apache#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping apache#67721's exact-N-per-Dag semantics. On run_after ties the window's (id DESC) tiebreak deterministically keeps the latest-inserted row, where the old per-branch LIMIT picked arbitrarily; the outer ORDER BY leaves tied rows in unspecified order, same as the union. All columns are selected inline (no join-back). dag_runs_limit is now NonNegativeInt, matching limit/offset. Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s same rows and order returned (291/291 id-set match, total_entries unchanged); selection can differ from the union only on run_after ties, where the new pick is the newer run Trade-off: the window sorts all runs of the page's Dags, so a page of the 50 highest-run-count Dags (~619K rows) costs ~2.0s with a 58MB external sort (vs the union's 0.25-12s bistable plan) — linear and deterministic instead of the 30-60s+ cold-page catastrophe. The window-function shape is portable (same ANSI syntax as the pre-apache#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com>
pickfire
added a commit
to pickfire/airflow
that referenced
this pull request
Oct 8, 2026
…ables The per-Dag UNION ALL branches introduced in apache#67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-apache#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping apache#67721's exact-N-per-Dag semantics. On run_after ties the window's (id DESC) tiebreak deterministically keeps the latest-inserted row, where the old per-branch LIMIT picked arbitrarily; the outer ORDER BY leaves tied rows in unspecified order, same as the union. All columns are selected inline (no join-back). dag_runs_limit is now NonNegativeInt, matching limit/offset. Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s same rows and order returned (291/291 id-set match, total_entries unchanged); selection can differ from the union only on run_after ties, where the new pick is the newer run Trade-off: the window sorts all runs of the page's Dags, so a page of the 50 highest-run-count Dags (~619K rows) costs ~2.0s with a 58MB external sort (vs the union's 0.25-12s bistable plan) — linear and deterministic instead of the 30-60s+ cold-page catastrophe. The window-function shape is portable (same ANSI syntax as the pre-apache#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com>
pierrejeambrun
pushed a commit
to pickfire/airflow
that referenced
this pull request
Oct 13, 2026
…ables The per-Dag UNION ALL branches introduced in apache#67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-apache#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping apache#67721's exact-N-per-Dag semantics. On run_after ties the window's (id DESC) tiebreak deterministically keeps the latest-inserted row, where the old per-branch LIMIT picked arbitrarily; the outer ORDER BY leaves tied rows in unspecified order, same as the union. All columns are selected inline (no join-back). dag_runs_limit is now NonNegativeInt, matching limit/offset. Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s same rows and order returned (291/291 id-set match, total_entries unchanged); selection can differ from the union only on run_after ties, where the new pick is the newer run Trade-off: the window sorts all runs of the page's Dags, so a page of the 50 highest-run-count Dags (~619K rows) costs ~2.0s with a 58MB external sort (vs the union's 0.25-12s bistable plan) — linear and deterministic instead of the 30-60s+ cold-page catastrophe. The window-function shape is portable (same ANSI syntax as the pre-apache#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com>
pierrejeambrun
pushed a commit
that referenced
this pull request
Oct 14, 2026
…ables (#74401) * Fix /ui/dags recent-runs query degradation on large, skewed dag_run tables The per-Dag UNION ALL branches introduced in #67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping #67721's exact-N-per-Dag semantics. On run_after ties the window's (id DESC) tiebreak deterministically keeps the latest-inserted row, where the old per-branch LIMIT picked arbitrarily; the outer ORDER BY leaves tied rows in unspecified order, same as the union. All columns are selected inline (no join-back). dag_runs_limit is now NonNegativeInt, matching limit/offset. Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s same rows and order returned (291/291 id-set match, total_entries unchanged); selection can differ from the union only on run_after ties, where the new pick is the newer run Trade-off: the window sorts all runs of the page's Dags, so a page of the 50 highest-run-count Dags (~619K rows) costs ~2.0s with a 58MB external sort (vs the union's 0.25-12s bistable plan) — linear and deterministic instead of the 30-60s+ cold-page catastrophe. The window-function shape is portable (same ANSI syntax as the pre-#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. Co-Authored-By: Claude Code <noreply@anthropic.com> * Add tests for recent-runs limit and run_after tie-breaking Per review: one test asserts a Dag with more runs than dag_runs_limit returns exactly the N newest in descending run_after order; one test pins the (run_after DESC, id DESC) tie-break — with two runs tied on run_after, the later-inserted run (higher id) is the deterministic pick, which is the one spot the window query's output differs from the old per-branch union. Co-Authored-By: Claude Code <noreply@anthropic.com> * Remove newsfragment per review Co-Authored-By: Claude Code <noreply@anthropic.com> * Trim redundant comments in the recent-runs tests Co-Authored-By: Claude Code <noreply@anthropic.com> * Keep dag_runs_limit as plain int The NonNegativeInt validation is a contract change unrelated to the regression this PR fixes — out of scope here. Co-Authored-By: Claude Code <noreply@anthropic.com> * Drop the explanatory comment above the ranked query Co-Authored-By: Claude Code <noreply@anthropic.com> --------- Co-authored-by: Claude Code <noreply@anthropic.com>
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 18, 2026
…ed dag_run tables (#74401) (#74966) * Fix /ui/dags recent-runs query degradation on large, skewed dag_run tables The per-Dag UNION ALL branches introduced in #67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping #67721's exact-N-per-Dag semantics. On run_after ties the window's (id DESC) tiebreak deterministically keeps the latest-inserted row, where the old per-branch LIMIT picked arbitrarily; the outer ORDER BY leaves tied rows in unspecified order, same as the union. All columns are selected inline (no join-back). dag_runs_limit is now NonNegativeInt, matching limit/offset. Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s same rows and order returned (291/291 id-set match, total_entries unchanged); selection can differ from the union only on run_after ties, where the new pick is the newer run Trade-off: the window sorts all runs of the page's Dags, so a page of the 50 highest-run-count Dags (~619K rows) costs ~2.0s with a 58MB external sort (vs the union's 0.25-12s bistable plan) — linear and deterministic instead of the 30-60s+ cold-page catastrophe. The window-function shape is portable (same ANSI syntax as the pre-#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. * Add tests for recent-runs limit and run_after tie-breaking Per review: one test asserts a Dag with more runs than dag_runs_limit returns exactly the N newest in descending run_after order; one test pins the (run_after DESC, id DESC) tie-break — with two runs tied on run_after, the later-inserted run (higher id) is the deterministic pick, which is the one spot the window query's output differs from the old per-branch union. * Remove newsfragment per review * Trim redundant comments in the recent-runs tests * Keep dag_runs_limit as plain int The NonNegativeInt validation is a contract change unrelated to the regression this PR fixes — out of scope here. * Drop the explanatory comment above the ranked query --------- (cherry picked from commit 1f70ead) Co-authored-by: Ivan Tham <pickfire@riseup.net> Co-authored-by: Claude Code <noreply@anthropic.com>
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 20, 2026
…ed dag_run tables (#74401) (#74966) * Fix /ui/dags recent-runs query degradation on large, skewed dag_run tables The per-Dag UNION ALL branches introduced in #67721 plan as backward index scans of the global idx_dag_run_run_after with a Filter on dag_id. On a large, skewed dag_run table this is catastrophic for "cold" Dags — those whose last run is months old or that never ran: the scan must walk essentially every row of every *other* Dag before reaching that branch's rows (measured on a 3.8M-row metadata DB: 3,800,261 rows removed by filter, ~2.3s for a single cold branch, 30-60s per page — gateway 504s), because at default statistics the planner misestimates cold dag_ids (they fall outside the MCV list) and bets the backward scan hits LIMIT matches quickly. Hot pages are affected too, bistably: the identical statement was observed flipping 22.8ms <-> 141s across a single ANALYZE. Replace the 50-branch union with one page-scoped window query: row_number() OVER (PARTITION BY dag_id ORDER BY run_after DESC, id DESC) over WHERE dag_id IN (page's dag ids), filtered to rn <= dag_runs_limit. This restores the pre-#67721 access path (per-Dag probes of idx_dag_run_dag_id + one sort + WindowAgg), bounded regardless of where each Dag's runs sit in time or what the planner estimates, while keeping #67721's exact-N-per-Dag semantics. On run_after ties the window's (id DESC) tiebreak deterministically keeps the latest-inserted row, where the old per-branch LIMIT picked arbitrarily; the outer ORDER BY leaves tied rows in unspecified order, same as the union. All columns are selected inline (no join-back). dag_runs_limit is now NonNegativeInt, matching limit/offset. Measured end-to-end on a dockerized copy of a 3.8M-row production metadata DB (PostgreSQL 15.14, default statistics, 74%-cold first page): GET /ui/dags?limit=50&dag_runs_limit=14: 31.2s -> 0.13-0.64s same rows and order returned (291/291 id-set match, total_entries unchanged); selection can differ from the union only on run_after ties, where the new pick is the newer run Trade-off: the window sorts all runs of the page's Dags, so a page of the 50 highest-run-count Dags (~619K rows) costs ~2.0s with a 58MB external sort (vs the union's 0.25-12s bistable plan) — linear and deterministic instead of the 30-60s+ cold-page catastrophe. The window-function shape is portable (same ANSI syntax as the pre-#67721 rank() query and the existing row_number usage in common/db/assets.py), so the sqlite/mysql/mssql matrix is unaffected. * Add tests for recent-runs limit and run_after tie-breaking Per review: one test asserts a Dag with more runs than dag_runs_limit returns exactly the N newest in descending run_after order; one test pins the (run_after DESC, id DESC) tie-break — with two runs tied on run_after, the later-inserted run (higher id) is the deterministic pick, which is the one spot the window query's output differs from the old per-branch union. * Remove newsfragment per review * Trim redundant comments in the recent-runs tests * Keep dag_runs_limit as plain int The NonNegativeInt validation is a contract change unrelated to the regression this PR fixes — out of scope here. * Drop the explanatory comment above the ranked query --------- (cherry picked from commit 1f70ead) Co-authored-by: Ivan Tham <pickfire@riseup.net> Co-authored-by: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While reviewing #67242, we found that several existing UI queries are also slower than expected on large dag_run tables (verified with ~
5MDagRuns). Adding new indexes would be a larger change (a migration), so this PR focuses on reusing the existing indexes. If there's a better direction, I'd appreciate your input.Changes (per query)
Screenshots
Before
After
cc @pierrejeambrun
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.