fix: Skip dataset-triggered dags without SerializedDagModel by leossantos · Pull Request #63546 · apache/airflow

leossantos · 2026-03-13T17:11:08Z

Summary

DagModel.dags_needing_dagruns could treat dataset-scheduled DAGs as ready for a new run when they had DatasetDagRunQueue rows but no SerializedDagModel row in the same evaluation window. The dataset timetable condition was never evaluated for those DAGs, yet they could still flow into dataset_triggered_dag_info, allowing premature dataset-triggered DagRuns.

This PR removes such dag_ids from the in-memory by_dag / dag_statuses maps until serialization exists. DatasetDagRunQueue ORM rows are not deleted here, so the scheduler can re-evaluate on a later heartbeat. DEBUG logging records skipped DAGs (missing serialization) and, when applicable, satisfied dataset conditions. Docstring updated accordingly.

Tests: two TestDagModel cases cover missing SerializedDagModel (single and multiple DAGs), DEBUG log expectations, and assert dataset_dag_run_queue row counts unchanged after dags_needing_dagruns.

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)
Cursor

boring-cyborg · 2026-03-13T17:11:12Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

airflow/jobs/scheduler_job_runner.py

airflow/models/dag.py

kaxil · 2026-03-24T22:36:23Z

A couple of meta-questions:

This PR targets v2-11-stable. The dag.py guard against missing SerializedDagModel is a real fix for a race condition. Does this same issue exist on main? If so, the fix should land there too (or first).
The diagnostic logging ([DEBUG DATASETS] prefix, N+1 lazy loads, per-loop queries) reads like temporary instrumentation added while investigating a specific incident. Is the intent to merge this into the stable branch permanently, or was this meant to be a local debugging patch? If it is meant to be permanent, the N+1 issues and log levels need to be addressed first — this would regress scheduler performance under load.

leossantos · 2026-03-26T19:44:26Z

@kaxil Thanks for the review.

Re: main — yes, the same structural issue exists on main (asset-based code path). We plan a separate PR against main with a manual port; this PR targets 2.11.x only. Happy to follow whichever merge order you prefer.

Re: logging — the [DEBUG DATASETS] / per-DAG re-query instrumentation was for our deployment debugging, not intended to merge as-is. The updated PR will keep only the SerializedDagModel guard + tests (+ newsfragment); any remaining log line in the fix will be DEBUG only. Inline comments (joinedload if needed, DDRQ preservation test, minor cleanups) will be addressed in the next push.

leossantos · 2026-03-27T18:04:29Z

I am creating the port forward for Airflow 3

leossantos · 2026-03-27T18:19:36Z

The port forwarding PR for main is #64322

leossantos · 2026-03-30T17:52:37Z

Friendly ping on this one. It's ready for a workflow run whenever a maintainer has a moment. Thank you!

leossantos · 2026-03-30T19:18:34Z

Several tests were failing because the DAGs were not being properly serialized. I've updated the test cases to ensure DAG serialization is handled correctly, which resolved the failures. Can you trigger the CI again?
I tested locally

airflow/models/dag.py

newsfragments/63546.bugfix.rst

tests/models/test_dag.py

Lee-W

a few nits, but overall looks good

airflow/models/dag.py

leossantos · 2026-03-31T12:27:26Z

@Lee-W Thanks for the suggestions! I've applied all of them. The PR is ready for another look.

leossantos · 2026-03-31T12:29:22Z

I got an error in documentations step, but I didn't understand how to solve this issue
https://github.com/apache/airflow/actions/runs/23762501024/job/69325975248

leossantos · 2026-03-31T13:29:11Z

I've addressed the feedback from the latest review.

tests/models/test_dag.py

leossantos · 2026-04-01T19:37:45Z

Solved the issue with the CI

Log which DAGs are selected as dataset-triggered (with ADRQ timestamp ranges) and log successful DagRun creation with dag_id, exec_date, prev_exec, event count, and event URIs. This provides visibility into the scheduler's dataset trigger decisions for debugging premature trigger incidents. Made-with: Cursor

…sions Log the full context of dataset-triggered scheduling to debug premature trigger incidents: - P0: Log condition, DDRQ URIs, and count when dataset_condition is satisfied (INFO in dags_needing_dagruns) - P1: Warn on DDRQ/event mismatch when queued URIs have no matching DatasetEvent in the timestamp range (WARNING in _create_dag_runs_dataset_triggered) - P2: Include data_interval start/end in the DagRun creation log - P3: Log consumed event timestamps and source DAG/run_id (DEBUG) Made-with: Cursor

Made-with: Cursor

…l is missing DAGs with DDRQ entries but no corresponding SerializedDagModel were bypassing dataset condition evaluation in dags_needing_dagruns() and entering dataset_triggered_dag_info unchecked. This caused premature triggers with partial events when the DAG processor was mid-parse cycle. Now explicitly detects the mismatch and excludes those DAGs from the current scheduler loop. DDRQ entries are preserved so the DAG is re-evaluated on the next heartbeat (~5s) once serialization completes. Made-with: Cursor

…DAGs This change ensures that the `missing_from_serialized` variable is deleted after its entries have been processed, preventing potential memory leaks and maintaining cleaner state management within the DAG model.

Drop [DEBUG DATASETS] instrumentation from SchedulerJobRunner and DagModel dataset-readiness loop; inline timetable dataset_condition where it is only used once.

Log the DDRQ-without-serialization case at debug and remove the [DEBUG DATASETS] prefix; drop redundant del of missing_from_serialized. Tests capture DEBUG, match the new message, and assert dataset_dag_run_queue rows remain after dags_needing_dagruns.

…ing_dagruns

Split DagModel and DatasetDagRunQueue inserts and flush after DagModel so foreign-key order matches production DB constraints in TestDagModel.

Ensure dag_maker writes SerializedDagModel so dags_needing_dagruns queries match scheduler expectations for dataset-triggered DAGs.

Co-authored-by: Wei Lee <weilee.rx@gmail.com>

dags_needing_dagruns now logs that dags are not found in serialized_dag instead of the older DDRQ/SerializedDagModel wording.

… remove associated newsfragment

leossantos changed the title ~~feat(scheduler): add INFO logging for dataset-triggered DagRun creation~~ feat(dag): Add control to remove non serialized dags from by_dag Mar 24, 2026

kaxil added this to the Airflow 2.11.3 milestone Mar 24, 2026

kaxil reviewed Mar 24, 2026

View reviewed changes

leossantos force-pushed the leo-santos/addtional-scheduler-fix branch from 5682993 to 872c56c Compare March 26, 2026 20:35

leossantos marked this pull request as ready for review March 27, 2026 18:03

leossantos requested review from XD-DENG and ashb as code owners March 27, 2026 18:03

leossantos mentioned this pull request Mar 27, 2026

fix(scheduler): skip asset-triggered dags without SerializedDagModel #64322

Merged

1 task

leossantos changed the title ~~feat(dag): Add control to remove non serialized dags from by_dag~~ feat(dag): Skip dataset-triggered dags without SerializedDagModel Mar 27, 2026

eladkal added the type:bug-fix Changelog: Bug Fixes label Mar 27, 2026

leossantos changed the base branch from v2-11-stable to v2-11-test March 30, 2026 13:20

leossantos force-pushed the leo-santos/addtional-scheduler-fix branch from a3787f9 to 7afbb29 Compare March 30, 2026 18:29

Lee-W reviewed Mar 31, 2026

View reviewed changes

airflow/models/dag.py Show resolved Hide resolved

newsfragments/63546.bugfix.rst Outdated Show resolved Hide resolved

tests/models/test_dag.py Outdated Show resolved Hide resolved

Lee-W reviewed Mar 31, 2026

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

airflow/models/dag.py Outdated Show resolved Hide resolved

Lee-W changed the title ~~feat(dag): Skip dataset-triggered dags without SerializedDagModel~~ fix: Skip dataset-triggered dags without SerializedDagModel Mar 31, 2026

leossantos requested review from Lee-W and kaxil March 31, 2026 18:03

Lee-W reviewed Apr 1, 2026

View reviewed changes

tests/models/test_dag.py Outdated Show resolved Hide resolved

Lee-W approved these changes Apr 2, 2026

View reviewed changes

leossantos force-pushed the leo-santos/addtional-scheduler-fix branch from 1392035 to 614b3ed Compare April 2, 2026 13:38

leossantos and others added 17 commits April 8, 2026 19:18

chore(scheduler): Add [DEBUG DATASETS] tag to logs

5fbf772

Made-with: Cursor

test(models): add tests for DDRQ entries without SerializedDagModel

e427302

chore(scheduler): remove dataset debug logging

f59cec3

Drop [DEBUG DATASETS] instrumentation from SchedulerJobRunner and DagModel dataset-readiness loop; inline timetable dataset_condition where it is only used once.

chore(datasets): debug-log when dataset condition passes in dags_need…

d089870

…ing_dagruns

docs: add bugfix newsfragment and dags_needing_dagruns dataset docstring

36105b1

test(dag): persist DagModel before DatasetDagRunQueue in unit tests

684fd2c

Split DagModel and DatasetDagRunQueue inserts and flush after DagModel so foreign-key order matches production DB constraints in TestDagModel.

test(dag): pass serialized=True in dataset dags_needing_dagruns tests

2c70fa3

Ensure dag_maker writes SerializedDagModel so dags_needing_dagruns queries match scheduler expectations for dataset-triggered DAGs.

Apply suggestions from code review

7f25115

Co-authored-by: Wei Lee <weilee.rx@gmail.com>

style(dag): normalize line endings in DDRQ serialized-DAG guard block

3c67b9f

test(dag): align DDRQ missing-serialized log assertions with dag.py

8561190

dags_needing_dagruns now logs that dags are not found in serialized_dag instead of the older DDRQ/SerializedDagModel wording.

test(dag): assert full DDRQ missing-serialized log line in caplog

e3198c9

chore: change missing serialized DAG log level from debug to info and…

e0425b5

… remove associated newsfragment

leossantos force-pushed the leo-santos/addtional-scheduler-fix branch from db58324 to e0425b5 Compare April 8, 2026 22:18

Conversation

leossantos commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg bot commented Mar 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaxil commented Mar 24, 2026

Uh oh!

leossantos commented Mar 26, 2026

Uh oh!

leossantos commented Mar 27, 2026

Uh oh!

leossantos commented Mar 27, 2026

Uh oh!

leossantos commented Mar 30, 2026

Uh oh!

leossantos commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

leossantos commented Mar 31, 2026

Uh oh!

leossantos commented Mar 31, 2026

Uh oh!

leossantos commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

leossantos commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

leossantos commented Mar 13, 2026 •

edited

Loading

leossantos commented Mar 30, 2026 •

edited

Loading

leossantos commented Mar 31, 2026 •

edited

Loading