Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions airflow-core/adr/lang-sdk/0008-control-flow-constructs.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ Proposed.
2. **A branch selects a task, not a string.** A case is the task reference the SDK already handed back, and the value on the wire is that task's task_id, so the compiler checks the candidate exists and no label has to be kept in step with it.
3. **No default case.** `BranchPythonOperator` has none to serialize, and a one-sided `If` whose condition is false follows nothing at all.
A decider returning a ref that is not one of the declared cases is a run-time error the SDK raises, a narrower check than `skip_all_except`, which only rejects a task_id missing from the whole Dag.
4. **Triggering a Dag run is an ordinary DSL task**, it is pure DSL purpose instead of a new runtime.
4. **Triggering a Dag run is an ordinary task that the SDK's own runtime runs.** It has no handler.
The runtime sends the `TriggerDagRun` request itself and follows `TriggerDagRunOperator` for every option it offers, including `wait_for_completion` and `deferrable`.
5. **Grouping keeps Python's semantics**: a scope offering the same task and nesting methods as the Dag, prefixing each task_id with the group id (`prefix_group_id`),
and can be ordered against a task or another group, as `group1 >> group2` does in Python.

Expand Down Expand Up @@ -89,7 +90,12 @@ That operator defaults to skipping every task in its downstream closure and igno
a single-ref return cannot express that, and no Lang SDK offers it for now.
An author needing several paths together puts them behind one task, or gates each with its own condition.
This limitation is accepted rather than open.
- **No Lang SDK needs a deferral mechanism for now** to offer `deferrable` or `wait_for_completion`, because the trigger task runs in Python.
- **Each Lang SDK runtime implements the trigger task**, so a native Dag needs no Python worker.
`deferrable` needs no trigger written in that language: the runtime defers to the standard provider's `DagStateTrigger`, which runs in the Python triggerer, and the task keeps its queue, so it resumes in the same runtime with `next_method` set.
- **The runtime reads the config the trigger task needs from its environment**, since it cannot read `airflow.cfg`.
It reads `AIRFLOW__API__BASE_URL`, `AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE` and `AIRFLOW__TRIGGERER__QUEUES_ENABLED`, and an unset one takes Python's fallback.
The coordinator exports them from Airflow's config (#73842).
- **The trigger's arguments are not Jinja-rendered**, because no Python code runs the task. A value is sent as the author wrote it.
- **Nothing extra reaches the Dag JSON**, which carries no branch-candidate field at all. A ref is a task_id by the time the decision is sent, so each SDK stores its cases and nothing else.
- **A group edge needs one base type per SDK** that both a task and a group satisfy, since either can sit at the end of an edge.
Python already has it: `TaskGroup(TaskGroupMixin, DAGNode)` (`task-sdk/src/airflow/sdk/definitions/taskgroup.py:96`) and every operator inherit `DependencyMixin` (`.../definitions/_internal/mixins.py:35`), where `set_upstream` and `set_downstream` live.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,99 @@ there. Set it once on the Dag and each task inherits it:
``queue`` on a task wins over the Dag's. See :ref:`typescript-sdk/coordinator-config` for the
``queue_to_coordinator`` entry that sends that queue to the coordinator.

Conditional branching
~~~~~~~~~~~~~~~~~~~~~

``dag.if`` takes a task whose handler returns a boolean, and names the task each outcome runs:

.. code-block:: typescript

const condition = dag.task("has_rows", async ({ rows }: { rows: number }) => rows > 0);
const gated = condition({ rows: extracted });

dag.if(gated).then(loaded).else(reportedEmpty);

The condition is an ordinary task, so it is declared, typed and wired like any other, and the
compiler checks that its handler really returns a boolean. ``else`` is optional: a one-sided
condition skips its own branch when the condition fails and follows nothing.

A guarded task takes no argument for the control edge, because a condition's boolean decides whether
the task runs rather than what it runs on. Read a value from the condition with
``getClient().getXCom``.

The side not taken is skipped when the run reaches it, and stays skipped if you clear it later. Only
the branches named here are skipped, so a task that several branches converge on still runs — unlike
Python's ``@task.branch``, which skips every immediate downstream it did not follow.

Multi-way branching
~~~~~~~~~~~~~~~~~~~

``dag.switch`` is the multi-way form: a task whose handler returns one of the cases it is given.

.. code-block:: typescript

const decider = dag.task("pick_path", async ({ rows }: { rows: number }) =>
rows > 1000 ? handleLong : handleShort,
);
const picked = decider({ rows: extracted });

dag.switch(picked).case(handleLong).case(handleShort);

A case is the task reference itself, so the compiler checks the candidate exists and renaming a
handler cannot silently rewire a Dag. The task's own value is the chosen task's id, which a
downstream task can read from its XCom.

There is no default case. A decider that returns anything outside its cases fails the task, naming
what it chose and what it could have chosen.

Exactly one case is selected. Python's branch callable may return a list of task ids, and no language
SDK offers that yet: put the paths that run together behind one task, or gate each with its own
condition.

Triggering another Dag
~~~~~~~~~~~~~~~~~~~~~~

``dag.triggerDagRun`` starts another Dag's run, as a task of this one:

.. code-block:: typescript

const trigger = dag.triggerDagRun({
taskId: "trigger_downstream",
dagId: "downstream_etl",
waitForCompletion: true,
});

trigger.after(loaded);

There is nothing to call: the task takes no TypeScript arguments, so it hands back the reference
directly. A second options object carries the task's own spec, as ``dag.task`` takes one.

The task runs in the TypeScript runtime, like the Dag's other tasks, and inherits the Dag's queue. It
behaves as ``TriggerDagRunOperator`` does. ``waitForCompletion`` polls the run every ``pokeInterval``
seconds until it reaches one of ``allowedStates`` or ``failedStates``. With ``deferrable`` as well, the
task defers to ``DagStateTrigger`` instead of holding a worker slot, and resumes in this runtime when
the run finishes. ``DagStateTrigger`` runs in the Python triggerer, so the triggerer needs the standard
provider installed.

The task pushes the ``trigger_run_id`` XCom, and the task's "Triggered DAG" link opens the run it
started.

The task follows these config options, with Python's fallback when one is unset:

- ``[api] base_url`` is the base of the "Triggered DAG" link. It falls back to ``/``.
- ``[operators] default_deferrable`` is the default of ``deferrable``. It falls back to ``false``.
- ``[triggerer] queues_enabled`` decides whether the deferred trigger gets the task's queue. It falls
back to ``false``, so the trigger gets none.

Some of what ``TriggerDagRunOperator`` does is not offered:

- ``logical_date`` and ``run_after``. The triggered run's logical date is the time the task triggers
it, as in Python when neither is set.
- Jinja. Values are sent as written, so ``{{ ds }}`` in ``conf`` reaches the triggered run as that
literal string.
- OpenLineage parent injection (``openlineage_inject_parent_info``). The runtime does not add the
parent task's OpenLineage details to ``conf``.

``new Dag`` and ``dag.task`` both take a trailing spec of Airflow options:
``{ schedule: "@daily", tags: ["etl"] }`` for the Dag, ``{ retries: 2, retryDelay: 30 }`` for a task.

Expand Down
66 changes: 66 additions & 0 deletions ts-sdk/adr/0002-native-dag-interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,72 @@ data is the wiring object itself — `summarize({ north: extractNorth(), south:
does. This matches `Before`/`After` in the Go SDK's native Dag interface, spelled to TypeScript
convention.

### Conditional branching: `if` and `else`

`dag.if(condition)` is TypeScript's spelling of the construct
[`airflow-core/adr/lang-sdk/0008`](../../airflow-core/adr/lang-sdk/0008-control-flow-constructs.md)
names after the host language's control flow:

```ts
const gated = dag.task("has_rows", async ({ rows }: { rows: number }) => rows > 0)({
rows: extracted,
});

dag.if(gated).then(loadIfReady).else(loadFallback);
```

**The condition is a task reference, not a task id and a function.** `dag.if` takes a `TaskRef` the
Dag already handed back, whose handler's return type the compiler checks is `boolean`. So nothing
depends on the task's id or on its function name, and a condition is declared, named and typed the
same way every other task is. The same reasoning that makes a *case* a reference in that ADR's
decision 2 applies to the condition itself.

**A `then` chain is a thenable, and is guarded rather than avoided.** An object with a callable
`then` is a *thenable*: were the object `dag.if` returns to reach an `await`, the runtime would hand
its `then` a resolve function where a task reference belongs. Two things contain that. `.then(...)`
returns an object carrying only `.else`, so nothing past the first step is awaitable at all; and
`.then` rejects a function argument by naming the cause, so an author who does await it reads "this
builds a branch, drop the await" rather than a type error about references.

**A branch is a real branch to Airflow.** The control edges serialize as ordinary order-only edges
and carry no branch-candidate field, as that ADR's consequences require. The condition task is
serialized with `_can_skip_downstream`, and it writes the `skipmixin_key` XCom alongside the skip, so
clearing a skipped branch re-skips it the way a Python `@task.branch` does rather than running the
side the condition rejected.

A one-sided `if` is a branch with one candidate — it skips `then` and follows nothing — rather than a
`ShortCircuitOperator`, which would also skip the whole downstream closure and ignore trigger rules.
A guarded task takes no argument for the control edge: a condition's boolean is a signal, not data.

### Multi-way branching: `switch` and `case`

`dag.switch(decider)` follows
[`airflow-core/adr/lang-sdk/0008`](../../airflow-core/adr/lang-sdk/0008-control-flow-constructs.md)
decision 2 without divergence: a case **is** the reference the SDK handed back, not a label kept in
step with one. The decider is one too.

```ts
const decider = dag.task("pick_path", async ({ rows }: { rows: number }) =>
rows > 1000 ? handleLong : handleShort,
);
const picked = decider({ rows: extracted });

dag.switch(picked).case(handleLong).case(handleShort);
```

An earlier draft selected a case by a string label the author writes, on the grounds that a handler's
function name does not survive bundling. That concern does not apply: a `TaskRef` carries the task's
own id, which the SDK fixed when the task was declared and esbuild never touches. Selecting by
reference keeps the compiler checking that a candidate exists, which a label cannot.

The cases chain, as they do in Go. `case` reads the candidate list when the task runs rather than
when it is declared, which is what lets the chain follow the `dag.switch(...)` call; and unlike a
condition's `then`, `case` is not a thenable trap, so nothing has to be guarded here.

**No default case**, per decision 3, and **exactly one case is selected**, the limitation that ADR
records for every Lang SDK. A branch with no case at all decides nothing, and is rejected when the
Dag is read.

## Consequences

- One authoring surface (`dag.task()` plus its factory) covers the graph and each task's arguments,
Expand Down
64 changes: 64 additions & 0 deletions ts-sdk/src/coordinator/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ import type {
GetXCom,
SetXCom,
GetConnection,
SkipDownstreamTasks,
TriggerDagRun,
GetDagRunState,
GetDag,
ConnectionResult as WireConnectionResult,
} from "./protocol.js";

Expand Down Expand Up @@ -82,6 +86,15 @@ export interface CoordinatorClient extends TaskClient {
* output fails the task, while one that pushed null binds null.
*/
getXComEntry(opts: GetXComOpts): Promise<XComEntry>;
/**
* Trigger a Dag run. `"already_exists"` is the one refusal a trigger task
* handles itself; any other error throws.
*/
triggerDagRun(msg: Omit<TriggerDagRun, "type">): Promise<"triggered" | "already_exists">;
/** The state of a Dag run, as `GetDagRunState` reports it. */
getDagRunState(dagId: string, runId: string): Promise<string>;
/** Whether a Dag is paused. */
isDagPaused(dagId: string): Promise<boolean>;
}

export function createCoordinatorClient(
Expand Down Expand Up @@ -202,6 +215,57 @@ export function createCoordinatorClient(
await rpc("SetXCom", null, msg, () => undefined, "throw");
},

// ---- Control flow ----

async skipDownstreamTasks(taskIds: readonly string[]): Promise<void> {
// The supervisor treats an empty list as "skip nothing", and sending it
// would cost a round trip to say so.
if (taskIds.length === 0) return;
const msg: SkipDownstreamTasks = { type: "SkipDownstreamTasks", tasks: [...taskIds] };
await rpc("SkipDownstreamTasks", null, msg, () => undefined, "throw");
},

// ---- Dag runs ----

async triggerDagRun(msg: Omit<TriggerDagRun, "type">) {
const request: TriggerDagRun = { type: "TriggerDagRun", ...msg };
logs?.debug("TriggerDagRun request");
const frame = await comm.request(request);
const err = parseFrameError(frame);
// Sent as an ErrorResponse body, not a frame error; see
// `ActivitySubprocess._handle_trigger_dag_run`.
if (err?.code === "DAGRUN_ALREADY_EXISTS") return "already_exists";
if (err) {
logs?.warning("TriggerDagRun failed", { error: err.code });
throw new Error(`TriggerDagRun failed: ${err.code}`);
}
return "triggered";
},

async getDagRunState(dagId: string, runId: string): Promise<string> {
const msg: GetDagRunState = { type: "GetDagRunState", dag_id: dagId, run_id: runId };
const state = await rpc(
"GetDagRunState",
"DagRunStateResult",
msg,
(body) => body!.state as string,
"throw",
);
return state!;
},

async isDagPaused(dagId: string): Promise<boolean> {
const msg: GetDag = { type: "GetDag", dag_id: dagId };
const paused = await rpc(
"GetDag",
"DagResult",
msg,
(body) => body!.is_paused === true,
"throw",
);
return paused!;
},

// ---- Connections ----

async getConnection(connId: string): Promise<ConnectionResult | null> {
Expand Down
9 changes: 9 additions & 0 deletions ts-sdk/src/coordinator/protocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import type {
DagFileParseRequest,
ErrorResponse,
DagFileParsingResult,
DeferTask,
RetryTask,
SucceedTask,
TaskState,
Expand All @@ -47,6 +48,7 @@ export type {

export type {
DagFileParsingResult,
DeferTask,
RetryTask,
SucceedTask,
TaskState,
Expand All @@ -64,6 +66,12 @@ export type {
GetXCom,
SetXCom,
GetConnection,
SkipDownstreamTasks,
TriggerDagRun,
GetDagRunState,
DagRunStateResult,
GetDag,
DagResult,
} from "../generated/supervisor.js";

// -------- Frames from supervisor --------
Expand All @@ -81,6 +89,7 @@ export type MsgFromSupervisor =

export type RuntimeTaskState = WithRequiredType<TaskState>;
export type RuntimeRetryTask = WithRequiredType<RetryTask>;
export type RuntimeDeferTask = WithRequiredType<DeferTask>;
export type RuntimeSucceedTask = WithRequiredType<SucceedTask> & {
task_outlets: NonNullable<SucceedTask["task_outlets"]>;
outlet_events: NonNullable<SucceedTask["outlet_events"]>;
Expand Down
22 changes: 20 additions & 2 deletions ts-sdk/src/coordinator/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,18 @@ import {
asMsgFromSupervisor,
SUPERVISOR_API_VERSION,
type RuntimeDagFileParsingResult,
type RuntimeDeferTask,
type RuntimeRetryTask,
type RuntimeSucceedTask,
type RuntimeTaskState,
type StartupDetails,
} from "./protocol.js";
import { getArgNames } from "../sdk/arg-names.js";
import { bundleDags, bundleDagTaskIds, type Bundle } from "../sdk/bundle.js";
import { bundleDags, bundleDagTaskIds, getBundleTrigger, type Bundle } from "../sdk/bundle.js";
import { finalizeDag } from "../sdk/dag.js";
import { SERIALIZATION_VERSION } from "../generated/dag-schema-fields.js";
import { computeRelativeFileloc, serializeDag } from "./serde.js";
import { runTriggerDagRun } from "./trigger-runner.js";
import { runInTaskScope, type TaskContext } from "../sdk/task.js";
import type { JsonValue } from "../sdk/client-types.js";

Expand Down Expand Up @@ -195,6 +197,8 @@ export async function startCoordinator(
await sendSupervisorResponse(firstFrame.id, response, comm, runtimeLogs);
if (response.type === "SucceedTask") {
runtimeLogs.info("Task succeeded", { task_id: body.ti.task_id });
} else if (response.type === "DeferTask") {
runtimeLogs.info("Task deferred", { task_id: body.ti.task_id });
}
} else {
const errMsg = `First frame must be DagFileParseRequest or StartupDetails, got ${body.type}`;
Expand Down Expand Up @@ -338,8 +342,22 @@ async function handleTask(
logs: LogChannel,
clientLogs: LogChannel,
signal: AbortSignal,
): Promise<RuntimeSucceedTask | RuntimeRetryTask | RuntimeTaskState> {
): Promise<RuntimeSucceedTask | RuntimeRetryTask | RuntimeTaskState | RuntimeDeferTask> {
const ti = details.ti;
const trigger = getBundleTrigger(bundle, ti.dag_id, ti.task_id);
if (trigger) {
const ctx = buildContext(details, signal);
const client = createCoordinatorClient(comm, ctx, clientLogs);
const fail = (message: string) => {
logs.error("Task failed", { task_id: ctx.taskId, error: message });
return buildFailureResponse(details, message);
};
try {
return await runTriggerDagRun(details, trigger, client, logs, ctx.signal, fail);
} catch (err) {
return fail((err as Error).message ?? String(err));
}
}
const handler = bundle.getTaskHandler(ti.dag_id, ti.task_id);

if (!handler) {
Expand Down
Loading