Skip to content

Speed up TaskGroup.topological_sort with int-indexed projected sweep - #67288

Merged
shahar1 merged 8 commits into
apache:mainfrom
shahar1:taskgroup-topo-sort-projection
May 27, 2026
Merged

shahar1 merged 8 commits into
apache:mainfrom
shahar1:taskgroup-topo-sort-projection

Conversation

@shahar1

@shahar1 shahar1 commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replaces the modified-Kahn body of TaskGroup.topological_sort (and the mirror in SerializedTaskGroup) with a projected sweep:

  • Each child's per-task upstream_task_ids is projected onto sibling-level integer indices once up front (_project_child_deps).
  • The sweep (_sweep_projection) then runs against the projection with a bytearray-backed emission flag - per-edge work happens once per sort instead of once per outer-loop pass.

Emission order is identical to the previous implementation. The existing order-sensitive tests (test_topological_sort1/2, test_topological_nested_groups, test_topological_group_dep) cover the contract.

Benchmark

Min of five runs, N=2000 children, ms per call. Source: https://gist.github.com/shahar1/9c61dc9f34f7e77cd29cfb9d67af7ceb

Shape main PR speedup
chain (forward-declared) 1.39 0.75 1.84x
diamond (root → many → sink) 2.20 0.86 2.56x
independent (no deps) 0.65 0.35 1.84x
layered (4-layer full bipartite) 230.96 67.15 3.44x
nested (depth-3, walked across all groups) 17.78 4.77 3.73x
rev-chain (adversarial reverse insertion) 894.04 114.16 7.83x

The 7.8x on rev-chain comes from the projection precompute alone (one projection vs N projections); algorithmic complexity is unchanged and remains O(N²) on that shape until the follow-up PR.

The nested row times a recursive walk across all 39 nested groups (depth=3, ~222 tasks each); the root-only timing was dominated by setup overhead and was previously labelled ~noise in this table.

Test plan

  • task-sdk/tests/task_sdk/definitions/test_taskgroup.py + test_dag.py — 145 passed (18 new shape-correctness cases)
  • airflow-core/tests/unit/utils/test_task_group.py — 20 passed (1 new serialization round-trip case)
  • airflow-core/tests/unit/serialization/ (task-group / topological filter) — 4 passed
  • prek run mypy-task-sdk — passed
  • prek run mypy-airflow-core — passed
  • prek run --from-ref upstream/main --stage pre-commit — passed

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.7)

Generated-by: Claude Code (Opus 4.7) following the guidelines

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants