Repository navigation
Make asset event batching opt-in - #68517
Conversation
c6a5e34 to
4d94b18
Compare
|
Early feedback while CI is running: Can you add some documentation with a very simple example to airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst? Note for reference, this is related to Slack discussion in https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1778258939480149 |
Done |
12fb085 to
527d8f2
Compare
jscheffl
left a comment
There was a problem hiding this comment.
Besides the comment nit I think this is a good solution to fix the problem raised in #53896 and most part of #56750 in my view. (in #56750 part 3 is a bug whereas any background is missing allowing a triage, so maybe this need either more evidence or should be closed).
But besides the positive discussions in Slack would like to have sone Asste Event experts additional opinion, mostly @uranusjr (alternatively @Lee-W / @ashb).
|
@renat-sagut looks like we have a conflict can you resolve |
|
@renat-sagut I've removed the Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. If you have questions, an Apache Airflow maintainer — a real person — is happy to help. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
Dag authors need batching choices to survive serialization without changing existing constructor calls. Align the feature with current scheduler queue handling and the reviewed release target.
Existing compound asset schedules must remain valid after reparsing. Changing batching modes must also preserve queued partition events and let users inspect the progress of each pending run.
The docs spelling check rejects "rollups", which is not in the wordlist, so the PR documentation build failed. Refer to RollupMapper instead of using the plural.
Merging pending partitioned runs when batching is enabled is a separate concern from making batching opt-in, and its scheduler locking is the riskiest part of the change. Keep this PR to the opt-in default and track the merge logic separately.
Asset schedules need to express their event requirements consistently during parsing and scheduling. The pending-run view should show the latest attempt when no specific run is selected.
2d39264 to
718893c
Compare
The cases have not changed since the harness landed in apache#69308, so they still encode the six-type policy. apache#74589 narrowed airflow-core to `significant` only -- "never for an ordinary bug fix, feature, improvement or doc change" -- which leaves the two cases that expect a feature newsfragment failing an agent for answering correctly. Flipping both would leave every case expecting "no", which an agent that always declines would pass, so two changes that do warrant a significant note keep the other direction covered: the asset-triggered run default from apache#68517 and the async Postgres driver default from apache#68496, both taken from newsfragments already on main.
Asset-triggered Dags currently combine queued asset events into a single run. This PR makes one run per event the default and adds
batch_asset_eventstoAssetTriggeredTimetable,PartitionedAssetTimetable, andAssetOrTimeScheduleso authors can opt into batching.The new
[scheduler] batch_asset_eventssetting defaults toFalse. An explicit timetable argument overrides it. Setting it toTrueand reparsing Dags restores the previous behavior; serialized timetables without the field retain batching until they are reparsed.With batching disabled, partitioned events get separate pending runs for each mapped partition. Non-partitioned run creation respects
max_active_runsand the scheduler's creation budget, retaining queued and catchup events until subsequent passes can consume them.Multi-asset AND expressions and partition rollups require
batch_asset_events=True. Parsing rejects those combinations when batching is disabled, preventing pending runs whose conditions can never be satisfied. The examples, documentation, and significant-change note explain this compatibility requirement.Regression coverage includes configuration overrides, SDK/core serialization and legacy payloads, repeated partition keys, simultaneous event timestamps, bounded backlog draining, catchup history, and Dag parsing.
Related #53896
Was generative AI tooling used to co-author this PR?
Generated-by: CodeWhale and Codex (GPT-6) following the guidelines