[hussein-awala]

Clearing a deferred task, or marking it failed or succeeded, cancels its trigger and runs on_kill() for up to [triggerer] on_kill_timeout (30 s by default). The supervisor re-sends the id in every sync until the trigger finishes (L656-L661), and the runner cancels the task again on each, about once per second. The second CancelledError lands inside on_kill(), where a branch written for shutdown swallows it (L1845-L1855), so cleanup such as deleting a pod stops after about a second, possibly leaving the resource running. Record when each trigger was cancelled, and cancel it again only after on_kill_timeout plus a 5 s margin.

related: #76209

No new configuration, database or protocol change. Only delivered cancels are recorded, so one arriving before the trigger task exists applies on the next re-send. The id is forgotten when the trigger finishes, so a reused id can be cancelled. The supervisor never moves an id between cancelling_triggers and releasing_triggers, so one record serves both.

A cancelled trigger now counts in the triggers.running and triggerer.capacity_left gauges while on_kill() runs; claiming new triggers counts database rows and is unaffected. A task that swallows its first cancellation is cancelled again after 35 s by default instead of about a second. The 5 s margin covers logging and cleanup() around on_kill(). Shutdown is unaffected.

#73702 (Glue) and #74358 (BigQuery) rely on on_kill() finishing. v3-4-test has the same cancel_triggers(); v3-3-test has only to_cancel and needs a small manual change.

A later PR planned in #76209 wakes the runner on trigger activity (up to about 20 syncs/s); without this fix, event traffic there cuts on_kill() off after about 50 ms, so this should merge first. #76218 touches the same cancel_triggers()/cleanup_finished_triggers() code, replacing the cleanup scan with a done-callback queue; whichever merges later is rebased and keeps the forget-on-finish step.

In a runner-level benchmark driving the real TriggerRunner methods, with a trigger whose on_kill() needs 3.0 s and a cancel re-sent every 0.25 s:

Metric	Before	After
on_kill() completed	no	yes
on_kill() start to task end	0.50 s	3.00 s
Syncs that carried the cancel	2	12
Create 500 triggers (control)	156.6 ms	157.4 ms

Before, the second cancel aborted on_kill() about 0.25 s in and the next poll saw the task end. After, the cancel is re-sent longer only because on_kill() now completes; only the first is delivered. Medians of five rounds, alternating which side runs first; these paths issue no SQL. Cancel rows matched every round; the control moved +0.5%, within noise. Local measurements, not end-to-end latency.

test_repeated_cancel_syncs_do_not_abort_on_kill re-sends the cancel while a real run_trigger() task is held in on_kill() by an event the test releases. Others cover redelivery after the grace period for both queues, a cancel arriving before its trigger task exists, and a reused id. Every new test fails against the previous logic.

Validation:

• Focused triggerer and trigger-model tests: 210 passed, 1 skipped on SQLite.
• PostgreSQL and MySQL were not run locally; this change has no database code.
• Regular and manual static checks passed on the full diff.
• What breeze verify lists passed on a local merge of the nine PRs opened together under Improve triggerer performance (AI-assisted deep dive) #76209: core DB tests of the API, Always, CLI, Core, Other and Serialization groups on SQLite (12,254 run, no failures), core non-DB tests (5,582 passed) and the apache-airflow documentation build.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

🤖 Generated with Claude Code

[fat-catTW]

Thanks for the PR

[seanghaeli]

Looks good. Nothing else needed here, just a note for my own trigger work, since on_kill can run for up to 30s after this PR, I shouldn't mark a trigger's status as "recovered" when it's in the process of being cancelled.