Repository navigation
Description
Description
The latest run of a Dag is picked in three different ways across these views.
| Where | How the latest run is picked |
|---|---|
Dashboard Failed Dags count (ui/dashboard.py dag_stats) |
logical_date DESC, runs without a logical date skipped |
Dags list last_dag_run_state filter and last_run_* sorting, and GET /api/v2/dags (common/db/dags.py generate_dag_with_latest_run_query) |
max(id) |
Dags list latest run shown on cards and in the table, Dag page header (latest_dag_runs, latest_run) |
run_after DESC |
I'd like to propose using run_after DESC, id DESC in all three, in two steps.
Step 1. Add a (dag_id, run_after) index on dag_run (#74481)
Picking the latest run by run_after needs an index in that order. dag_run has indexes on dag_id and on run_after, but none that keeps the runs of one Dag in run_after order, so a per-Dag lookup either sorts all runs of the Dag or scans idx_dag_run_run_after and filters on dag_id. #74401 ran into the same thing for the Dags list recent runs. This step changes no behavior and already speeds up GET /ui/dags/{dag_id}/latest_run for Dags whose runs are old.
Step 2. Pick the latest run by run_after DESC, id DESC in all three (#75803)
run_afteris never null, unlikelogical_date, which is null for manual runs without a logical date and for asset-triggered runs.- Most of what users see already uses it. The places that would change are the dashboard count and the
max(id)filter and sort. _check_last_n_dagruns_failed, which decides whether to pause a Dag after consecutive failures, already orders byrun_after DESC, id DESC.
This changes what GET /api/v2/dags returns for last_dag_run_state, dag_run_state, dag_run_start_date_* and dag_run_end_date_*, and for last_run_state and last_run_start_date sorting, whenever the newest-created run is not the latest by run_after. The main case is a backfill over a past interval.
The Grid orders runs by the timetable's run_ordering, which is a separate, pluggable concept and is out of scope here.
Use case/motivation
"Latest run" is one concept, but each view defines it differently, so a user can see one run as the latest on a card, filter Dags by the state of another, and count failed Dags from a third. The dashboard Failed Dags count and the Dags list it links to can disagree, and the list can show one latest run while filtering and sorting by another. Backfills over a past interval and manual runs without a logical date make this visible.
Can we settle on run_after as the definition of a Dag's latest run here?
Related issues
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
[-]Pick a Dag's latest run by run_after everywhere in the UI and REST API[/-][+]Pick a Dag's latest run by run_after consistently in the UI and REST API[/+]TensorDevLJ commentedon Oct 9, 2026
Thanks for the detailed breakdown. I'd be interested in picking up Step 2 — switching the three latest-run lookups (dashboard
dag_stats,generate_dag_with_latest_run_query, and confirminglatest_dag_runs/get_latest_run_infoalready match) torun_after DESC, id DESC, once #74481 lands.Is Step 2 still open, or do you already have it in progress? Happy to take a different angle if so — just didn't want to duplicate work.
yuseok89 commentedon Oct 11, 2026
Thanks for the interest! Step 2 isn't ready to pick up yet. It depends on #74481 landing first, and on maintainers confirming that
run_afteris the right definition of a Dag's latest run. Once both are settled I'd like to work on it myself, as I noted when opening this issue.