Skip to content

Group and filter Dags by folder on the Dags page - #68544

Merged
pierrejeambrun merged 3 commits into
apache:mainfrom
RabahAmrouche05:dags-folder-grouping
Oct 2, 2026
Merged

pierrejeambrun merged 3 commits into
apache:mainfrom
RabahAmrouche05:dags-folder-grouping

Conversation

@RabahAmrouche05

@RabahAmrouche05 RabahAmrouche05 commented Jun 14, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds a folder navigation tree to the Dags page. The tree is built from each
Dag's relative_fileloc directory, and selecting a folder filters the Dag list
to the Dags that live in it (and its subfolders).

Why

The Dags page only offered a flat list. In deployments with many Dags spread
across subfolders of the dags folder (e.g. team_a/etl/…, team_b/ml/…) there
was no way to browse or filter by folder.

How

  • Backend — a server-side relative_fileloc_prefix filter on GET /ui/dags
    (strict folder match, so team_a never catches team_alpha) and a new
    GET /ui/dags/folders endpoint returning the distinct folders of all readable
    Dags. Filtering is done server-side so the result stays correct across
    pagination, sorting and the other filters.
  • Frontend — a collapsible folder tree on the Dags page that drives the
    filter via a dag_folder search param, with the tree hierarchy reconstructed
    client-side from the folder paths.

Dags located at the bundle root contribute no folder and appear under "All Dags".

Tests: backend (folders endpoint + filter, incl. the team_a / team_alpha
edge case) and frontend (buildFolderTree unit tests + DagFolderTree
component tests).


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines


Important

🛠️ Maintainer triage note for @RabahAmrouche05 · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @pierrejeambrun is waiting on you:

  • 3 unresolved review thread(s) from @pierrejeambrun need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

Automated triage — may be imperfect; a maintainer takes the next look.

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default labels Jun 14, 2026
@boring-cyborg

boring-cyborg Bot commented Jun 14, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@jscheffl

Copy link
Copy Markdown
Contributor

Can you please add a screenshow or small video recording of how it looks like?

@RabahAmrouche05

Copy link
Copy Markdown
Contributor Author

Thanks for the review @jscheffl !

Here's what the feature looks like:

  • Screenshot — the example DAG folder structure on disk (team_a/etl/…, team_b/ml/…).
Capture d’écran 2026-06-17 à 15 49 30
  • Video — navigating the new folder tree on the Dags page: expanding/collapsing folders, selecting a folder to filter the list (server-side, so it stays correct across pagination), and going back to "All Dags".
Enregistrement.de.l.ecran.2026-06-17.a.16.03.29.mov

I also removed the newsfragment as you suggested.

One more note: a main merge had left the generated OpenAPI TS client out of sync (the GetDagFolders types were lost during the merge resolution), which was failing the static checks. I've regenerated the client, so that should be green now.

@RabahAmrouche05
RabahAmrouche05 requested a review from jscheffl June 17, 2026 17:50
@potiuk
potiuk marked this pull request as draft June 22, 2026 06:35
@pierrejeambrun

pierrejeambrun commented Jun 22, 2026 •

Copy link
Copy Markdown
Member

PR is marked as draft, are you planning additional work before being ready for review?

@RabahAmrouche05

Copy link
Copy Markdown
Contributor Author

Hi @pierrejeambrun No further work planned, it's ready for review. I've addressed @jscheffl 's feedback (added a screenshot + a short video of the feature, and removed the newsfragment).

@RabahAmrouche05
RabahAmrouche05 marked this pull request as ready for review June 26, 2026 20:29

@pierrejeambrun pierrejeambrun left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, that's cool!

Tested and working as expected, a few suggestions though.

Always-rendered tree — even deployments with all Dags at the bundle root get the 280px sidebar (just "All Dags" + an empty message). Slight clutter for the flat case; consider hiding the panel when there are no folders. Or even better, a visual toggle to hide/display this sidebar.

CI need fixing (conflicts, rebasing should also fix the failing check)

@pierrejeambrun pierrejeambrun changed the title UI: Group and filter Dags by folder on the Dags page Group and filter Dags by folder on the Dags page Jun 29, 2026
@pierrejeambrun pierrejeambrun added this to the Airflow 3.3.1 milestone Jun 29, 2026
@potiuk potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 8, 2026
@bbovenzi bbovenzi modified the milestones: Airflow 3.3.1, Airflow 3.4.0 Jul 9, 2026
@pierrejeambrun pierrejeambrun removed the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 20, 2026
@pierrejeambrun

pierrejeambrun commented Jul 20, 2026 •

Copy link
Copy Markdown
Member

Need to fix the conflicts before we can merge.

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just had a chat with Jed, very good comment from him is that this doesn't take into account multi bundle deployment.

Meaning that dags coming from different bundle will be merged under the same tree. All dags will appear as if they were collocated under the same directory. Which probably isn't what the user expect.

Would it be possible to update the PR so when there are multiple bundles (has to be returned in the DagFolderCollectionResponse payload, for each folder basically) the bundle is displayed in the UI as the top level entry and then the folder tree appears under each bundle. When there is only one bundle keep the current UI.

@RabahAmrouche05

Copy link
Copy Markdown
Contributor Author

Thanks @pierrejeambrun and @jedcunningham, good catch on the multi-bundle case! I updated the PR.

The folders endpoint now returns the bundle with each folder, since a folder path is relative to its bundle and can exist in more than one. When there are several bundles the tree shows the bundle first and the folders under it, so two bundles with the same path don't get mixed together. With a single bundle it stays flat like before.

Selecting a folder now uses its bundle too (with the existing bundle_name filter), so team_a/etl in two bundles points to the right dags.

Also rebased on main to fix the conflicts. Added tests for both (same path in two bundles staying separate, and the bundle grouping in the tree).

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIce thanks for updating the PR. Can you add / update the screen recording in the case of multi bundles configuration.

@RabahAmrouche05
RabahAmrouche05 force-pushed the dags-folder-grouping branch 2 times, most recently from d7818d7 to 1707631 Compare August 2, 2026 11:56
@RabahAmrouche05

Copy link
Copy Markdown
Contributor Author

Thanks @bbovenzi @pierrejeambrun, switched the tree to the chakra TreeView component. No more onClick on divs, and we get keyboard navigation and the aria attributes for free.

I also fixed two things I hit while testing with a real multi bundle setup:

  • Alignment: folders with children were shifted to the right compared to their siblings, because only they render an expand chevron. Leaf folders now render a hidden placeholder (same icon, visibility hidden + aria-hidden) so both line up on the same column.

  • Stale dags: the folders endpoint now filters them out (is_stale == false), to match the dag list which hides stale dags by default. Before that, a bundle that stopped being parsed kept its folders in the tree, and clicking one gave you an empty list.

Both are covered by tests.

Here is the setup I used, two bundles that both have a team_a/etl folder:

Capture d’écran 2026-08-01 à 23 42 31

And the tree in the UI, each bundle keeping its own folders:

Enregistrement.de.l.ecran.2026-08-01.a.23.41.10.mov

@pierrejeambrun pierrejeambrun left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay.

Looking nice! Thanks for the improvement.

A few small comments and I think we'll be good to merge!

@RabahAmrouche05
RabahAmrouche05 force-pushed the dags-folder-grouping branch 2 times, most recently from 2cbd0c4 to 4ee0848 Compare August 26, 2026 20:13
@RabahAmrouche05

Copy link
Copy Markdown
Contributor Author

Thanks @pierrejeambrun for the review! All three points are addressed:

  • Expanding from a URL: the tree state was only computed on the first render, when the folders are still loading, so it never expanded. It now re-applies the auto expansion once the selection resolves to real nodes, while keeping whatever the user expanded manually. You were right about the test too, it was passing the folders synchronously, so the new test renders with no folders first and then with them.

  • Long names: rows now take the full panel width and the label truncates with an ellipsis, so the highlight covers the whole row. The full name is still available as a tooltip.

  • Added a button to show/hide the tree, next to the sort and display controls. The choice is remembered in localStorage, so you can get the full width table back.

Also rebased on main.

@bbovenzi

Copy link
Copy Markdown
Contributor

Ok let's do one last rebase before merging?

Add a collapsible folder navigation tree to the Dags page, built from each Dag's
relative_fileloc directory. Selecting a folder filters the list to the Dags it
contains (and its subfolders) via a new server-side filter, so the result stays
correct across pagination, sorting and other filters.

Because relative_fileloc is relative to each bundle, folders are scoped to their
bundle: in multi-bundle deployments the bundle is shown as the top-level entry
with its folder tree nested underneath, so folders sharing the same path in
different bundles stay separate. Single-bundle deployments keep the flat tree.

Backend adds a relative_fileloc_prefix filter on GET /ui/dags and a new
GET /ui/dags/folders endpoint returning the distinct (bundle, folder) pairs of
all readable Dags. Dags at the bundle root contribute no folder and appear under
'All Dags'. The folder sidebar is hidden for flat deployments with no folders.

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just rebased the branch and fixed the conflicts.

Should be ready to go.

@pierrejeambrun
pierrejeambrun merged commit e148542 into apache:main Oct 2, 2026
100 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 2, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants