Skip to content

Default batch_asset_events to the constructed value when absent from serialized data - #75009

Closed
vatsrahul1001 wants to merge 1 commit into
mainfrom
fix-asset-batch-deserialize-default
Closed

vatsrahul1001 wants to merge 1 commit into
mainfrom
fix-asset-batch-deserialize-default

Conversation

@vatsrahul1001

Copy link
Copy Markdown
Contributor

What

Asset-triggered timetables deserialized a missing batch_asset_events key as True, but the field is opt-in and defaults to False (added in #68517) — or to True only when the asset condition requires batching. Any serialized DAG whose blob lacks the key (every DAG serialized before 3.4, and the window on upgrade before the dag-processor re-parses) therefore came back batching queued asset events into a single DAG run instead of one run per event.

Fix: only pass batch_asset_events when the key is present, so a missing key falls back to the same attrs factory default the constructor uses — False for a plain condition, True for AND/partition conditions that require batching. Applied at all three deserialize sites:

  • timetables/simple.py — AssetTriggeredTimetable, PartitionedAssetTimetable
  • timetables/assets.py — AssetOrTimeSchedule

A hardcoded False would be wrong because AND/partition timetables require batching; deferring to the factory keeps both cases correct.

Added a regression test (test_deserialize_without_batch_asset_events_key_uses_construct_default) covering plain (→False), AND (→True), and explicit-value round-trip.

Closes: #75007


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

@vatsrahul1001 vatsrahul1001 added the backport-to-v3-4-test Backport to v3-4-test label Oct 18, 2026
@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.4.0 milestone Oct 18, 2026
@vatsrahul1001
vatsrahul1001 force-pushed the fix-asset-batch-deserialize-default branch from f20ebcc to 170dc0c Compare October 18, 2026 16:20
…serialized data

Asset-triggered timetables deserialized a missing batch_asset_events key as True, but the field is opt-in and defaults to False (or to True only when the asset condition requires batching). Blobs that predate the field -- and the window on upgrade before the dag-processor re-parses a DAG -- therefore came back batching queued asset events into a single run instead of one run per event.

Pass batch_asset_events to the timetable only when the key is present, so a missing key falls back to the same attrs factory default the constructor uses: False for a plain condition, True for AND/partition conditions that require batching. Applied at all three deserialize sites (AssetTriggeredTimetable, PartitionedAssetTimetable, AssetOrTimeSchedule).

Closes: #75007
@vatsrahul1001
vatsrahul1001 force-pushed the fix-asset-batch-deserialize-default branch from 170dc0c to 8c6d303 Compare October 18, 2026 16:24
@vatsrahul1001

Copy link
Copy Markdown
Contributor Author

Closing — this is not a bug. The absent-key batch_asset_events=True fallback is the documented, intentional back-compat from #68517: "Already serialized timetables without this option retain their previous batching behavior until reparsed" (see airflow-core/newsfragments/68517.significant.rst and docs/authoring-and-scheduling/asset-scheduling.rst). Pre-#68517 behavior was to batch all queued events into one run; keeping True for un-reparsed blobs preserves that across a 3.3→3.4 upgrade until the dag-processor reparses the DAG. Defaulting to the factory value (False) here would instead create a per-event run burst on upgrade for backlogged asset DAGs — the regression #68517 was written to avoid. Apologies for the noise. Closing #75007 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-4-test Backport to v3-4-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Asset timetable deserializes batch_asset_events to True when key absent, contradicting the opt-in False default

1 participant