[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes - claude sonnet 4.6

Important Note

What "durable" and "deferrable" mean in the PR: there are two different reasons an operator can survive a crash, and they work in totally different ways. "Durable" means the operator's own code remembers where its job was and reconnects to it instead of starting over, and this operates using task state store in synchronous mode. "Deferrable" is running using a trigger, in asynchronous mode. The end result looks similar to a user (nothing gets redone), but one is a promise the operator has to keep and the other is just how the system already works for everyone.

What

Follow-up to the merged supports_durable_execution badge PR (#67236). Now that we have badge support, we should also have a few filters for ease of navigation for two related-but-distinct capabilities: "durable" (anything that survives a crash without redoing work: ResumableJobMixin/manual-marker operators, or deferrable ones) and "deferrable" (the narrower, deferrable-only subset). The registry had no deferrable detection at all before this change.

Current behaviour

The Registry only tracks supports_durable_execution (ResumableJobMixin or the manual __supports_durable_execution marker). Deferrable operators, a much more common capability across most cloud providers, have no signal anywhere in the pipeline, and there's no way to filter a provider's module list by either capability.

Proposed change

• Durable detection: is_durable_capable(cls) in dev/registry/extract_parameters.py walks the resolved execute() chain from cls, following both super().execute() and an explicit ParentClass.execute(self, context) call (the pattern task decorators like @task.agent/@task.kubernetes use), to check whether a class that declares the durable marker, or genuinely implements ResumableJobMixin, still preserves that behavior after subclassing. Whole-line comments are stripped before matching so a comment that merely mentions delegation isn't mistaken for real code.
• Deferrable detection: supports_deferrable(cls) added to dev/registry/extract_parameters.py. Checking for a deferrable constructor parameter isn't enough, since a subclass can inherit the parameter while overriding execute() with code that never reads it, and an operator that always defers unconditionally has no such parameter at all. This walks outward from execute() into helper methods it calls via self.<name>(...), and up the MRO through delegation, looking for an actual self.defer(...) call, a self.deferrable read, or a TaskDeferred raise. Code that's only reachable on a core older than the one being imported (e.g. a defer() fallback behind if AIRFLOW_V_3_3_PLUS: raise ..., as in the HITL and common.ai approval operators) is excluded, since it never actually runs on the core the extractor imports.
• Schema: supports_deferrable: bool = False added to ModuleContract, wired into make_entry() alongside supports_durable_execution (13 fields total, kept as an independent signal, not merged into the existing one).
• Frontend: two independent badges on module cards ("Durable", "Deferrable": teal/blue) shown per their own signal, since an operator can have either, both, or neither. Two filter toggles on a provider's module list: "Durable (includes deferrable)" and "Deferrable only".

Changes of Note

• The two filters are a deliberate union/subset pair, not independent booleans. "Durable" is supports_durable_execution OR supports_deferrable, checking it shows every deferrable operator plus every ResumableJobMixin/manual-marker operator, even ones that aren't deferrable. "Deferrable only" is supports_deferrable alone, never OR'd. This means an operator that is both durable and deferrable appears under both filters, which is intentional and not confusing in practice, since the two independent badges on the card explain why.
• The checkbox labels ("Durable (includes deferrable)" / "Deferrable only") were chosen specifically to make that asymmetric containment relationship (deferrable is a subset of durable, never the reverse) visible at the point of clicking, rather than relying on a tooltip or a caption someone could miss.
• is_deferrable inheriting a parent's __init__ unmodified still counts as deferrable (unlike the supports_durable_execution manual-marker case, which deliberately does not propagate to subclasses), since inheriting an unmodified constructor is a real functional guarantee the parameter still works, whereas inheriting a durable-execution claim says nothing about whether an overridden execute() still preserves it.
• This PR also widens is_durable_capable() itself, not just adds the new deferrable signal alongside it. That function's existing marker-lookup logic had three separate false-negative bugs (an MRO walk that lost track across multiple inheritance, a lookup that gave up on a class with no own execute override, and a name-mangling calculation that broke on classes with a leading underscore), all now fixed. This changes the already-shipped Durable badge's output for a few real operators, most notably EksPodOperator and GKEStartPodOperator, which were previously misreported as not durable-capable and now correctly show the badge. SparkKubernetesOperator also reports durable, correctly, but for a reason the detector doesn't actually check (see the docstring on _delegates_execute_to); its deferrable path never delegates via super() at all, and is durable only because KubernetesPodOperator's execute_sync/execute_async both independently call the pod-reattach mechanism.

Testing

• Built breeze to extract the provider data using: breeze registry extract-data

• Built the actual Eleventy site against this real data and confirmed in the rendered HTML: both badges appear on the right operator cards (e.g. Databricks' page shows both a "Durable" and "Deferrable" badge on DatabricksRunNowOperator), both filter checkboxes render with the intended labels, and the data-durable/data-deferrable attributes are correctly populated per the OR/non-OR logic on every module checked.

For current changes, this is the behaviour:

• Databricks: DatabricksSubmitRunOperator and DatabricksRunNowOperator are both durable and deferrable. DatabricksNotebookOperator, DatabricksSQLStatementsOperator, DatabricksTaskBaseOperator, DatabricksTaskOperator, DatabricksStartWarehouseOperator, DatabricksStopWarehouseOperator, and DatabricksSQLStatementsSensor are deferrable only. The rest of the provider's operators/hooks/sensors are neither.
• Apache Spark: SparkSubmitOperator is durable but not deferrable (no deferrable param).
• Google/BigQuery: BigQueryInsertJobOperator is both durable and deferrable, same pattern as Databricks.
• Amazon/Redshift: RedshiftDataOperator is both durable and deferrable.
• Snowflake: SnowflakeSqlApiOperator and SnowflakeNotebookOperator are both durable and deferrable.

Across the full dataset: 1995 modules total, 17 durable, 241 deferrable, 14 with both. The three durable-only entries (durable without a deferral path) are SparkSubmitOperator, AgentOperator, and the @task.agent decorator. The 14 that are both: EksPodOperator, GlueJobOperator, RedshiftDataOperator, LivyOperator, KubernetesPodOperator, SparkKubernetesOperator, DatabricksRunNowOperator, DatabricksSubmitRunOperator, BigQueryInsertJobOperator, GKEStartPodOperator, SnowflakeSqlApiOperator, SnowflakeNotebookOperator, and the @task.kubernetes / @task.kubernetes_cmd decorators.

Manual verification (breeze registry extract-data, generated dev/registry/modules.json)

Every operator raised across all review rounds, checked against the freshly generated data:

Operator	durable	deferrable	Verdict
EksPodOperator	True	True	Durable fix confirmed
GKEStartPodOperator	True	True	Durable fix confirmed
SparkKubernetesOperator	True	True	Durable fix confirmed (correct answer, coincidental reasoning per the _delegates_execute_to docstring note)
SnowflakeNotebookOperator	True	True	Durable via the delegation walk
KubernetesPodOperator	True	True	Baseline unchanged
AgentOperator	True	False	Baseline unchanged
HITLOperator	False	False	defer_for_approval fix confirmed
ApprovalOperator	False	False	Fix confirmed
HITLBranchOperator	False	False	Fix confirmed
HITLEntryOperator	False	False	Fix confirmed
LLMOperator	False	False	Fix confirmed
LLMBranchOperator	False	False	Fix confirmed
LLMSQLQueryOperator	False	False	Fix confirmed
LLMSchemaCompareOperator	False	False	Fix confirmed
LLMFileAnalysisOperator	False	False	Fix confirmed
DiscordWebhookOperator	False	False	No false-positive regression
HttpOperator	False	True	Real deferral still detected
DatabricksRunNowOperator	True	True	Matches PR description example
DatabricksSubmitRunOperator	True	True	Matches PR description example
KubernetesJobOperator	False	True	Correctly not durable (own Job path)
GKEStartJobOperator	False	True	Correctly not durable
SparkJDBCOperator	False	False	Correctly not durable (drops execute_resumable)

Some screenshots:

In this image, "Durable (includes deferrable)" is selected, and you can see both durable + deferrable (BigQueryInsertJobOperator) and just deferrable ones selected

"Deferrable only selection":

Operator like snowflake has only one operator with both deferrable and durable property, so its here:

Spark is one of few operators with only "durable", and no deferrable option:

What's next

Interactive click-through verification (actually toggling the checkboxes in a running browser to confirm filterModules() behaves correctly) hasn't been done -- only the generated HTML/data have been checked, not runtime JS behavior. Registry-wide (cross-provider) discovery -- "which are all the durable/deferrable operators across the whole registry," via a Pagefind search filter facet -- was discussed and deliberately deferred to a separate follow-up.

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[kaxil]

Round 5 wrap-up, so everything outstanding is in one place. All of it is verified against c2d04a7.

Blocking

1. The defer_for_approval thread, which I have just widened: it is nine classes across two providers, not five, and a fix scoped to defer_for_approval leaves the four standard HITL operators wrong.
2. flex-wrap: wrap on .capability-filter-toggles, which currently overflows 390px on every provider page that renders it.

Still open from round 4

The field-count nit. The stronger assertion in ef2d05c is an improvement, but the nit was the count in the name, and at HEAD there are now three hardcoded copies rather than two: the discover_classes_from_provider docstring ("all 13 Module fields"), test_has_all_13_fields, and test_all_13_fields_present with its 13-element required_fields list. Module is a @dataclass, so {f.name for f in fields(Module)} collapses all three and cannot drift. If you would rather keep the literal, say so and I will resolve the thread.

Satisfied, please resolve

These three are collapsed as outdated in the UI, so they are easy to miss, but they are done:

• The provider-version.njk tooltip. HEAD reads "Pass deferrable=True where the operator exposes it; some defer unconditionally", which is the fix.
• --color-teal-600. Gone from tokens.css, with no references left in main.css.
• The delegation-shapes thread. The rewrite covers all four shapes. I re-ran the detector at HEAD: EksPodOperator, GlueCrawlerOperator, and the Databricks task and warehouse operators all come back True, and DiscordWebhookOperator stays False.

Description

Two statements in the body no longer match the code:

• "checks for a deferrable constructor parameter, reusing the existing get_params_from_class() machinery". That detector is gone. HEAD walks execute() outward through self.<helper>() calls and up the MRO through super().execute(), looking for deferral tokens.
• "14 fields total". Module and ModuleContract both carry 13 at HEAD, which is also what both tests assert.

The "1779 modules total, 6 durable, 217 deferrable, 5 with both" line predates both detector rewrites and will move again once the badge question lands, so it is worth re-running at the end rather than now. The Databricks examples still hold; I re-checked the four listed as deferrable-only.

Not asking for either of these

Listing them so neither surfaces later as a surprise. The title= tooltips on the badges and on the new toggles are not keyboard-reachable, but that is a pattern this PR inherits rather than introduces. And the filter checkboxes have the same bfcache and form-restore exposure that moduleSearch already has today.

[kaxil]

Round-6 items all verified at 4c6aacf: the docstring note, the decorator capability fields (all 24 decorators resolve, @task.agent and @task.kubernetes now badge durable), and the three new tests each fail when the code they pin is reverted.

Two small things before merge, neither blocking. The description's aggregate counts (1779/6/217/5) and the "SparkSubmitOperator is the single durable-only case" line are out of date now that the numbers have settled (AgentOperator and @task.agent are durable-only too, and the delegation walk also flips SparkKubernetesOperator and SnowflakeNotebookOperator to durable, both correctly). Since this gets squash-merged under the body it is worth a refresh. Also, test_module_contract_accepts_legacy_modules_without_supports_deferrable asserts on the input dict because _validate returns its payload, so it passes even with the field deleted or its default flipped; ModuleContract.model_validate(payload).supports_deferrable is False is the binding form (the durable sibling has the same shape).

Optional follow-up material, not for this PR: the round-5 origin-resolution and depth-key fixes have no test that fails without them, and _execute_chain_calls_resumable still substring-matches unstripped source while the deferral path strips comments. The Durable badge tooltip ("Reconnects to an already-running job on retry") also predates this PR and does not describe AgentOperator or @task.agent, whose durable mode replays persisted step state rather than reconnecting to a job.

[amoghrajesh]

@kaxil Both done.

Description refreshed against a fresh breeze registry extract-data: 1995 modules, 17 durable, 241 deferrable, 14 both. Dropped the "single durable-only case" line; durable-only is SparkSubmitOperator, AgentOperator and @task.agent. Also added SnowflakeNotebookOperator to the Snowflake bullet and the verification table, and
the two Databricks warehouse operators to the deferrable-only list, which the old "the rest are neither" sentence got wrong.

Both legacy-module tests now assert on the parsed model rather than the input dict: ModuleContract.model_validate(payload).supports_deferrable is False, and the same for the durable sibling. Flipping either default to True fails them.

Taking the rest as follow-up: tests for the round-5 origin-resolution and depth-key
fixes, comment stripping in _execute_chain_calls_resumable, and the Durable badge
tooltip wording for the replay case.

[kaxil]

@kaxil Both done.

Description refreshed against a fresh breeze registry extract-data: 1995 modules, 17 durable, 241 deferrable, 14 both. Dropped the "single durable-only case" line; durable-only is SparkSubmitOperator, AgentOperator and @task.agent. Also added SnowflakeNotebookOperator to the Snowflake bullet and the verification table, and the two Databricks warehouse operators to the deferrable-only list, which the old "the rest are neither" sentence got wrong.

Both legacy-module tests now assert on the parsed model rather than the input dict: ModuleContract.model_validate(payload).supports_deferrable is False, and the same for the durable sibling. Flipping either default to True fails them.

Taking the rest as follow-up: tests for the round-5 origin-resolution and depth-key fixes, comment stripping in _execute_chain_calls_resumable, and the Durable badge tooltip wording for the replay case.

Feel free to merge after CI 🎉

[amoghrajesh]

The failing checks aren't related. Merging the PR.

[github-actions]

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 8a8bd52 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.