Skip to content

Add async Variable methods (aget, aset, adelete, akeys) - #72329

Merged
dabla merged 12 commits into
apache:mainfrom
dabla:feature/add-async-accessors-variable
Oct 5, 2026
Merged

dabla merged 12 commits into
apache:mainfrom
dabla:feature/add-async-accessors-variable

Conversation

@dabla

@dabla dabla commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Add async Variable methods (aget, aset, adelete, akeys) with tests

This PR adds async counterparts to all synchronous Variable methods.

What changed

Variable class (task-sdk/src/airflow/sdk/definitions/variable.py)

Four new async class methods:

Async method Sync equivalent Notes
Variable.aget Variable.get Awaits _async_get_variable; falls back to default (masked) on VARIABLE_NOT_FOUND
Variable.aset Variable.set Awaits _async_set_variable
Variable.adelete Variable.delete Awaits _async_delete_variable
Variable.akeys Variable.keys Awaits _async_get_variable_keys and returns the resolved list; unlike keys it is not lazy

Context helpers (task-sdk/src/airflow/sdk/execution_time/context.py)

This PR also adds the five helpers behind those methods. Each mirrors its sync twin:

  • _async_mask_and_deserialize_variable masks the raw value and, with
    deserialize_json, the decoded value through amask_secret.
  • _async_get_variable checks SecretCache, then asks each secrets backend:
    aget_variable when the backend has one, otherwise get_variable via
    asyncio.to_thread. ExecutionAPISecretsBackend.aget_variable reaches the
    Execution API via asend. An authoritative deny
    (AirflowSecretsBackendAccessDenied) is re-raised and never falls through to
    the next backend.
  • _async_set_variable warns when a worker-side secrets backend already holds
    the key (checked via asyncio.to_thread), sends PutVariable via asend,
    and invalidates the cache.
  • _async_delete_variable sends DeleteVariable via asend and invalidates
    the cache.
  • _async_get_variable_keys pages through GetVariableKeys requests via asend.

Tests

test_variables.py

TestAsyncVariables and TestAsyncVariableKeys mirror the sync test cases:

  • test_avar_get: plain and JSON-deserialized values; verifies the GetVariable request
  • test_avar_get_returns_default_when_not_found, test_avar_get_raises_when_not_found_without_default
  • test_avar_set: plain and JSON-serialized values; verifies the PutVariable payload
  • test_avar_delete: verifies DeleteVariable is sent via asend
  • test_akeys: prefix filtering (all / with prefix / empty)
  • test_akeys_paginates_when_results_exceed_page_size
  • test_akeys_raises_on_error_response
  • test_akeys_raises_on_unexpected_response_type

test_context.py

TestAsyncVariableContext tests the context helpers directly:

  • _async_get_variable: from the API, from a secrets backend, not found raises,
    a PERMISSION_DENIED from the Execution API raises and the next backend is never asked
  • masking: cache hit, plain value, JSON dict / list / string values, sensitive key, invalid JSON
  • _async_set_variable: plain value, JSON-serialized value, warning on a conflicting backend
  • _async_delete_variable: sends DeleteVariable
  • _async_get_variable_keys: prefix filtering, pagination, error, unexpected response

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    GitHub Copilot (Claude Sonnet 4.6), Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@dabla dabla self-assigned this Aug 31, 2026
Fix Variable.akeys: wrapping an async call inside lazy_object_proxy.Proxy
is invalid because await cannot be used inside a plain lambda. The method
now awaits _async_get_variable_keys directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dabla
dabla force-pushed the feature/add-async-accessors-variable branch from 93418a6 to 6c0e252 Compare August 31, 2026 14:43
@dabla dabla mentioned this pull request Sep 11, 2026
1 task done
@dabla dabla added this to the Airflow 3.4.0 milestone Sep 14, 2026

@amoghrajesh amoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes to avoid accidental merge, directionally good, but there's work needed.

dabla and others added 3 commits September 15, 2026 12:41
…ctly

Avoid importing and reassigning AsyncMock in every test; the fixture's
asend is already an AsyncMock, so tests can set return_value/side_effect
on it directly, per top-level import convention.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cMock imports

Same cleanup as test_variables.py: set return_value/side_effect directly
on the fixture's mock_supervisor_comms.asend instead of importing and
reassigning AsyncMock per test; move json import to the top of the file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dabla
dabla force-pushed the feature/add-async-accessors-variable branch from af48dc4 to f8be671 Compare September 15, 2026 12:09
@dabla
dabla requested a review from amoghrajesh September 16, 2026 06:43
dabla added a commit to dabla/airflow that referenced this pull request Sep 30, 2026
The error an iterated async task gets for a synchronous SDK call already
points at Variable.aget/aset, which apache#72329 adds; the class docstring
still said Variable had no async equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@amoghrajesh amoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking better, some comments.

dabla and others added 4 commits September 30, 2026 13:12
json is already imported at module level in context.py, and
_VARIABLE_KEYS_PAGE_SIZE is now imported once at the top of each test module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mirror the sync VariableAccessor masking tests for _async_get_variable so
_async_mask_and_deserialize_variable and the SecretCache hit path are covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dabla
dabla requested a review from amoghrajesh September 30, 2026 14:46
dabla added a commit to dabla/airflow that referenced this pull request Oct 1, 2026
The error an iterated async task gets for a synchronous SDK call already
points at Variable.aget/aset, which apache#72329 adds; the class docstring
still said Variable had no async equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@amoghrajesh amoghrajesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR body is wrong. It mentions "The four underlying async* functions were already implemented" and lists them as background. This PR adds all four, along with _async_mask_and_deserialize_variable, please update it.

Looks good otherwise with 2 nits.

dabla and others added 2 commits October 5, 2026 14:58
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
_async_get_variable re-raises AirflowSecretsBackendAccessDenied instead of
asking the next secrets backend. The sync twin is covered in test_secrets.py;
this pins the async one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dabla added a commit to dabla/airflow that referenced this pull request Oct 5, 2026
The error an iterated async task gets for a synchronous SDK call already
points at Variable.aget/aset, which apache#72329 adds; the class docstring
still said Variable had no async equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dabla
dabla merged commit 6785360 into apache:main Oct 5, 2026
108 checks passed
dabla added a commit to dabla/airflow that referenced this pull request Oct 8, 2026
The error an iterated async task gets for a synchronous SDK call already
points at Variable.aget/aset, which apache#72329 adds; the class docstring
still said Variable had no async equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants