Skip to content

Make asset event batching opt-in - #68517

Merged
vatsrahul1001 merged 33 commits into
apache:mainfrom
renat-sagut:batch-asset-events
Oct 12, 2026
Merged

vatsrahul1001 merged 33 commits into
apache:mainfrom
renat-sagut:batch-asset-events

Conversation

@renat-sagut

@renat-sagut renat-sagut commented Jun 13, 2026 •

Copy link
Copy Markdown
Contributor

Asset-triggered Dags currently combine queued asset events into a single run. This PR makes one run per event the default and adds batch_asset_events to AssetTriggeredTimetable, PartitionedAssetTimetable, and AssetOrTimeSchedule so authors can opt into batching.

The new [scheduler] batch_asset_events setting defaults to False. An explicit timetable argument overrides it. Setting it to True and reparsing Dags restores the previous behavior; serialized timetables without the field retain batching until they are reparsed.

from airflow.sdk import DAG, Asset, AssetTriggeredTimetable, PartitionedAssetTimetable

source = Asset("s3://bucket/source")

with DAG(
    dag_id="per-event",
    schedule=AssetTriggeredTimetable(assets=source, batch_asset_events=False),
):
    ...

with DAG(
    dag_id="per-partition-event",
    schedule=PartitionedAssetTimetable(assets=source, batch_asset_events=False),
):
    ...

With batching disabled, partitioned events get separate pending runs for each mapped partition. Non-partitioned run creation respects max_active_runs and the scheduler's creation budget, retaining queued and catchup events until subsequent passes can consume them.

Multi-asset AND expressions and partition rollups require batch_asset_events=True. Parsing rejects those combinations when batching is disabled, preventing pending runs whose conditions can never be satisfied. The examples, documentation, and significant-change note explain this compatibility requirement.

Regression coverage includes configuration overrides, SDK/core serialization and legacy payloads, repeated partition keys, simultaneous event timestamps, bounded backlog draining, catchup history, and Dag parsing.

Related #53896


Was generative AI tooling used to co-author this PR?
  • Yes — CodeWhale and Codex (GPT-6)

Generated-by: CodeWhale and Codex (GPT-6) following the guidelines

@renat-sagut
renat-sagut force-pushed the batch-asset-events branch 2 times, most recently from c6a5e34 to 4d94b18 Compare June 14, 2026 11:22
@jscheffl

Copy link
Copy Markdown
Contributor

Early feedback while CI is running: Can you add some documentation with a very simple example to airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst?

Note for reference, this is related to Slack discussion in https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1778258939480149

@renat-sagut

Copy link
Copy Markdown
Contributor Author

Early feedback while CI is running: Can you add some documentation with a very simple example to airflow-core/docs/authoring-and-scheduling/asset-scheduling.rst?

Note for reference, this is related to Slack discussion in https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1778258939480149

Done

@renat-sagut
renat-sagut force-pushed the batch-asset-events branch 2 times, most recently from 12fb085 to 527d8f2 Compare June 14, 2026 17:34
@renat-sagut
renat-sagut marked this pull request as ready for review June 14, 2026 17:38

@jscheffl jscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides the comment nit I think this is a good solution to fix the problem raised in #53896 and most part of #56750 in my view. (in #56750 part 3 is a bug whereas any background is missing allowing a triage, so maybe this need either more evidence or should be closed).

But besides the positive discussions in Slack would like to have sone Asste Event experts additional opinion, mostly @uranusjr (alternatively @Lee-W / @ashb).

@jscheffl jscheffl added this to the Airflow 3.3.0 milestone Jun 14, 2026
@potiuk potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jun 17, 2026
@vatsrahul1001

Copy link
Copy Markdown
Contributor

@renat-sagut looks like we have a conflict can you resolve

@potiuk

potiuk commented Jul 11, 2026

Copy link
Copy Markdown
Member

@renat-sagut I've removed the ready for maintainer review label because the next step here is yours: unresolved review thread(s) from @uranusjr awaiting your reply. It'll go back into the maintainer queue automatically once that's done — no need to re-add the label by hand. Next move: address the unresolved threads and reply to @uranusjr. There is no rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. If you have questions, an Apache Airflow maintainer — a real person — is happy to help. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@potiuk potiuk removed the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 11, 2026
renat-sagut and others added 18 commits October 11, 2026 19:56
Dag authors need batching choices to survive serialization without changing existing constructor calls. Align the feature with current scheduler queue handling and the reviewed release target.
Existing compound asset schedules must remain valid after reparsing. Changing batching modes must also preserve queued partition events and let users inspect the progress of each pending run.
The docs spelling check rejects "rollups", which is not in the wordlist, so the PR documentation build failed. Refer to RollupMapper instead of using the plural.
Merging pending partitioned runs when batching is enabled is a separate
concern from making batching opt-in, and its scheduler locking is the
riskiest part of the change. Keep this PR to the opt-in default and track
the merge logic separately.
Asset schedules need to express their event requirements consistently during parsing and scheduling. The pending-run view should show the latest attempt when no specific run is selected.
@vatsrahul1001
vatsrahul1001 merged commit 3ac1e4f into apache:main Oct 12, 2026
109 checks passed
namanjain24-sudo added a commit to namanjain24-sudo/airflow that referenced this pull request Oct 28, 2026
The cases have not changed since the harness landed in apache#69308, so they
still encode the six-type policy. apache#74589 narrowed airflow-core to
`significant` only -- "never for an ordinary bug fix, feature,
improvement or doc change" -- which leaves the two cases that expect a
feature newsfragment failing an agent for answering correctly.

Flipping both would leave every case expecting "no", which an agent that
always declines would pass, so two changes that do warrant a significant
note keep the other direction covered: the asset-triggered run default
from apache#68517 and the async Postgres driver default from apache#68496, both
taken from newsfragments already on main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conditional Dataset(Asset) Updates Are Grouped Together Instead of Creating Distinct DAG Runs

8 participants