Skip to content

TS SDK: embed the entrypoint source in the Dag bundle - #73127

Merged
jason810496 merged 1 commit into
apache:mainfrom
jason810496:feature/ts-sdk/bundle-entrypoint-source
Sep 19, 2026
Merged

jason810496 merged 1 commit into
apache:mainfrom
jason810496:feature/ts-sdk/bundle-entrypoint-source

Conversation

@jason810496

@jason810496 jason810496 commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Why

The shipped code region is minified, so it was nothing readable for Airflow to display for a natively authored TypeScript Dag so we need another section in the artifact to show the entry point raw source.

How

  • A packed bundle now carries the entry module as its author wrote it, in a region with its own byte range and SHA-256 digest in the layout header:

    //# airflowBundle={"code":{...},"metadata":{...},"source":{...}}
    //# airflowMetadata={...}
    /*# airflowSource
    <the entrypoint, readable>
    #*/
    <minified code>
    
  • Only the entry module will be included in the source, not the whole module graph behind it.

  • A block comment rather than the line comments the layout and metadata use.

  • The source range's length is declared rather than derivable from a newline, so the reader pins it by checking that the prescribed opener and closer sit where the declared range implies.

  • Rename the metadata dags mapping to task_handlers, which is what a TypeScript bundle currently provides: handlers for Dags declared elsewhere, not Dag definitions of its own.

  • The cross-language golden fixture is reworked. Its code region was the raw contents of a .ts fixture while its
    source region was a trivial export {};, which is backwards from what the packer emits.

    • It now carries a code region nobody could read and a source region that exercises both escape branches,
      so the Python reader is validated against bytes the TypeScript encoder actually produced.
    • The fixture is added to .prettierignore, which would otherwise reformat it and invalidate its digests.

Was generative AI tooling used to co-author this PR?

@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:e2e-tests area:task-sdk area:ts-sdk kind:documentation labels Sep 14, 2026
@jason810496
jason810496 marked this pull request as draft September 15, 2026 02:20
@jason810496
jason810496 force-pushed the feature/ts-sdk/bundle-entrypoint-source branch 3 times, most recently from 65b5ebd to 44b821d Compare September 17, 2026 05:22
@jason810496 jason810496 self-assigned this Sep 17, 2026
@jason810496
jason810496 marked this pull request as ready for review September 17, 2026 05:30
@jason810496
jason810496 force-pushed the feature/ts-sdk/bundle-entrypoint-source branch from 44b821d to 82dc7a8 Compare September 17, 2026 06:44

@pierrejeambrun pierrejeambrun left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice — LGTM thanks

@jason810496
jason810496 force-pushed the feature/ts-sdk/bundle-entrypoint-source branch from 82dc7a8 to 64dbd56 Compare September 18, 2026 03:45
@jason810496
jason810496 marked this pull request as draft September 18, 2026 12:57
A packed bundle now ships the entry module as its author wrote it, in a
`/*# airflowSource ... #*/` block comment with its own byte range and SHA-256
digest in the layout header. The shipped code region is minified, so it is not
the code anyone wrote and there was nothing readable for the Airflow UI to show
for a natively authored TypeScript Dag. Only the entry module is embedded, not
the module graph behind it; ADR-0006 declined multi-file source display for
mixed-language Dags.

A block comment, not the line comments the layout and metadata use, because the
entrypoint spans the lines it was written on. That makes line terminators legal
inside the region and leaves `*/` as the one sequence that must not appear: it
would end the comment where Node reads the file, putting the rest of the payload
into executable position while the layout still calls those bytes source and
both digests still match. The packer inserts a `\` between the two characters,
and escapes `*\` the same way so the transformation is reversible; the reader
reverses it and rejects a bundle carrying the sequence unescaped rather than
vouch for one.

The cross-language golden fixture now carries regions shaped like what the
packer ships, instead of the raw contents of a `.ts` fixture: a code region
nobody could read, and a source region that exercises both escape branches. The
escape scheme is a byte-level agreement between the TypeScript encoder and the
Python reader, and only an artifact the encoder actually produced can establish
it; the Python-side helper that escapes source for other tests is a
reimplementation and cannot. The fixture is also excluded from Prettier, which
would otherwise reformat it and invalidate its digests.

The metadata `dags` mapping is renamed to `task_handlers`, which is what a
TypeScript bundle actually provides: handlers for Dags declared elsewhere, not
Dag definitions of its own. The mapping stays keyed by Dag ID.

airflow_bundle_metadata_version stays at 1.0, redefined in place because the
TypeScript packing workflow is unreleased. A source-less bundle still fails
closed, on the now-required `source` layout section, before any version check
is reached.
@jason810496
jason810496 force-pushed the feature/ts-sdk/bundle-entrypoint-source branch from 64dbd56 to 50de2c4 Compare September 18, 2026 13:01
@jason810496
jason810496 marked this pull request as ready for review September 19, 2026 03:25
@jason810496
jason810496 merged commit 8cdb183 into apache:main Sep 19, 2026
207 of 208 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants