Skip to content

Pick a Dag's latest run by run_after in the API and dashboard - #75803

Open
yuseok89 wants to merge 8 commits into
apache:mainfrom
yuseok89:latest-run-by-run-after
Open

yuseok89 wants to merge 8 commits into
apache:mainfrom
yuseok89:latest-run-by-run-after

Conversation

@yuseok89

@yuseok89 yuseok89 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

The Dags list filters and sorting, GET /api/v2/dags and the dashboard Failed Dags count picked a Dag's latest run differently from the Dag cards and the Dag page, which use run_after. They now all pick it by run_after, so a backfill over a past interval or a manual run without a logical date no longer makes them disagree.

This uses the (dag_id, run_after) index from #74481. On 5M runs with 10% of the Dags dormant, a GET /api/v2/dags request filtered or sorted by the latest run went from 1.7 to 3.3 s to about 25 ms, and the dashboard stayed the same.

Before / After

Two Dags, one whose manual run without a logical date failed after an earlier scheduled run succeeded, and one whose run for an older interval failed after a newer scheduled run succeeded.

Before. The Failed Dags card shows 0, while the list it links to shows both Dags.

Screen.Recording.2026-10-28.at.11.27.08.AM.mov

After. The card shows 1, and the list shows only the Dag whose latest run failed.

Screen.Recording.2026-10-28.at.11.24.52.AM.mov

closes: #74482
related: #57492


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Claude Code (Opus 5.5)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label Oct 27, 2026

@fat-catTW fat-catTW left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for standardizing the latest-run definition.
Could we add coverage for two DagRuns with the same run_after?

@yuseok89

yuseok89 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! A tie now goes to the higher id, with a test in each place that picks the latest run. The Dag page also ordered by run_after only, so it now breaks ties by id as well.

@github-actions github-actions Bot added the fork ci required Gated by AIP-120: CI must pass in the contributor's fork. Managed by the fork CI reconciler. label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Fork CI: green. CI for the latest commit passed in your fork. The pull request is ready for maintainer review.

Pull requests from contributors run their CI in the contributor's own fork. GitHub Actions is free and has no minute limit on public forks, while the CI capacity of the Apache Software Foundation is shared by every ASF project and is finite.

Fork CI tested this branch merged with main at 5c9c4d85e8, as project CI would, 1 commits behind main when the run turned green.

Latest run: https://github.com/yuseok89/airflow/actions/runs/43782681524

How this works: CI in your fork.


This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated.

@github-actions github-actions Bot added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Nov 8, 2026
@github-actions github-actions Bot added ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. and removed ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. labels Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API fork ci required Gated by AIP-120: CI must pass in the contributor's fork. Managed by the fork CI reconciler. ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pick a Dag's latest run by run_after consistently in the UI and REST API

2 participants