Skip to content

Fix asset watcher triggers deleted on every dag-parsing loop#64625

Closed
vatsrahul1001 wants to merge 1 commit intomainfrom
fix/asset-watcher-trigger-hash-mismatch
Closed

Fix asset watcher triggers deleted on every dag-parsing loop#64625
vatsrahul1001 wants to merge 1 commit intomainfrom
fix/asset-watcher-trigger-hash-mismatch

Conversation

@vatsrahul1001
Copy link
Copy Markdown
Contributor

Summary

Fix asset watcher triggers being deleted and recreated on every dag-parsing loop.
encode_trigger wraps kwargs with BaseSerialization.serialize() (e.g., tuple() becomes
{"__type": "tuple", "__var": []}), which is correct for JSON-serializable DAG definitions.
However, add_asset_trigger_references needs raw Python kwargs because Trigger._decrypt_kwargs
returns native Python objects from the DB. This mismatch caused BaseEventTrigger.hash() to
produce different hashes for the same logical trigger, resulting in an infinite delete/recreate
cycle every parsing loop.
Replace encode_trigger with _get_raw_trigger_kwargs that unwraps any BaseSerialization-encoded
values back to native Python objects, ensuring hash consistency between DAG-defined and DB-stored
triggers. encode_trigger remains in use where it belongs — DAG serialization for JSON storage.

  • Yes (please specify the tool below)
    Claude
  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@TJaniF
Copy link
Copy Markdown
Contributor

TJaniF commented Apr 2, 2026

Tested with a local Kafka cluster and works great :)

__init__ py — airflow 2026-04-02 at 2 50 30 PM Airflow 2026-04-02 at 2 50 20 PM

@vatsrahul1001 vatsrahul1001 force-pushed the fix/asset-watcher-trigger-hash-mismatch branch from 12d8ca2 to 55960a6 Compare April 2, 2026 13:12
@vatsrahul1001 vatsrahul1001 requested a review from uranusjr April 2, 2026 13:16
Copy link
Copy Markdown
Contributor

@phanikumv phanikumv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add tests !

Copy link
Copy Markdown
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few comments but lgtm directionally but will defer to @uranusjr and @Lee-W for 2nd pass

@vatsrahul1001 vatsrahul1001 force-pushed the fix/asset-watcher-trigger-hash-mismatch branch from 6ac4d7a to 3f10aad Compare April 2, 2026 18:00
@vatsrahul1001 vatsrahul1001 added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label Apr 2, 2026
@vatsrahul1001 vatsrahul1001 marked this pull request as draft April 2, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:DAG-processing backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants