Add runtime DagRun note updates by hkc-8010 · Pull Request #69403 · apache/airflow

hkc-8010 · 2026-07-05T07:12:11Z

Add a Task SDK helper for runtime code to update the current Dag run note without writing directly to the metadata database.

The Execution API endpoint is scoped to the authenticated task instance and derives the target DagRun server-side. Runtime-authored notes are stored without a user attribution, while UI/public API edits continue to use the current user.

closes: #64890

Was generative AI tooling used to co-author this PR?

Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

pierrejeambrun

Overall looking good to me, a couple of questions though:

Automatic note at runtime will override and modify manually (user) notes on a run / TI.

Is that expected or shold we 'warn' and keep user note instead in this scenario?

pierrejeambrun

LGTM, this just needs conflict solving.

vatsrahul1001 · 2026-09-10T04:41:39Z

@hkc-8010 can you resolve conflicts here

hkc-8010 · 2026-09-10T07:32:23Z

Rebased on main. The only conflict was in execution_api/versions/__init__.py, where both sides had appended to the same version tuple, but resolving that as a straight union would have been wrong. This branch was cut when 2026-06-30 was the in-progress execution API version. It has been released since, and 2026-10-30 is in progress now, so registering a new endpoint under 2026-06-30 makes a released version advertise an endpoint it never shipped. I moved AddDagRunNoteUpdateEndpoint to v2026_10_30.py.

I also added versions/v2026_10_30/test_dag_run_notes.py for the previous-version coverage the versioning guide asks for. It fails if the change sits on a released version, so it pins the placement. Same shape as the existing test_callbacks.py.

The rest is docstrings. The route, client method and runtime helper now spell out that an empty note removes the note while a null note leaves it alone, since note: str | None reads the other way on its own.

jason810496

Thanks for adding the feature, LGTM overall.

Runtime code needs a supported Task SDK path for updating the current Dag run note without writing directly to the metadata database. Keeping the write scoped to the authenticated task instance preserves the Execution API isolation boundary while supporting callback use cases.

The triggerer message-type guard needs to record worker-only runtime operations explicitly. Dag run note updates require a task execution context and should not become part of the triggerer supervisor protocol.

The runtime note endpoint reimplemented the set/update/clear logic from patch_dag_run_note and diverged on clearing: a runtime clear left a dag_run_note row with NULL content, while the UI/public API deletes it. Make patch_dag_run_note's user attribution optional (user_id) and reuse it from the execution API so both paths behave the same, including removing the note row on clear. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The endpoint was registered under Version("2026-06-30"), which was the in-progress execution API version when this branch was cut. That version has since been released and 2026-10-30 is the in-progress one now, so the released 2026-06-30 spec was advertising an endpoint that release never shipped. Move the VersionChange to v2026_10_30.py and add the previous-version test the versioning guide asks for, which fails if the change sits on a released version. Also document that a null note leaves an existing note untouched while an empty note removes it, since the signature alone reads the other way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Skip the supervisor round-trip when the note is None, log when a runtime write replaces an attributed note, and rename the attribution test to say what it proves.

pierrejeambrun

LGTM, thanks

run.py now takes a scenario: its own Airflow base commit, bundle scope, AGENTS.md pointer and tasks. The metrics scenario is unchanged. The new dagrun-note scenario is Airflow at the parent of apache/airflow#69403 (runtime DagRun note updates). The change crosses the Task SDK runtime, the supervisor protocol and its versioned schema, the SDK HTTP client and generated models, the core Execution API with a Cadwyn version, and the generated TypeScript SDK. The task is to write an implementation plan from issue #64890. The plan is graded on the 11 parts of the real change it covers, a checklist written from the upstream diff before any trial ran. Upstream's file list scores 11/11, the issue text 0/11 and upstream's PR description 1/11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Run 5 in EXPERIMENTS.md. Haiku 5.5 built a 17-concept bundle of the Execution API and Task SDK runtime (verify 0 findings), then planned apache/airflow#69403 three times with the bundle and three times without. Plan score was 0.79 vs 0.76, inside the spread of each arm, and input tokens were 652k vs 642k. The bundle documents the parts the plans missed, the agent cited it, and still explored as much as without it. Two checklist items were widened after the first scoring (route and runtime-helper files), for both arms; every plan used valid files that upstream did not. Before widening the means were 0.64 vs 0.61. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hkc-8010 requested review from amoghrajesh, ashb, ephraimbuddy, jedcunningham and kaxil as code owners July 5, 2026 07:12

boring-cyborg Bot added area:API Airflow's REST/HTTP API area:dag-processor area:task-sdk area:ts-sdk kind:documentation labels Jul 5, 2026

potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 8, 2026

eladkal added this to the Airflow 3.4.0 milestone Jul 15, 2026

eladkal added the type:new-feature Changelog: New Features label Jul 15, 2026

jason810496 removed the area:ts-sdk label Jul 18, 2026

kaxil reviewed Jul 21, 2026

View reviewed changes

Comment thread task-sdk/src/airflow/sdk/execution_time/schema/versions/v2026_06_16.py Resolved

kaxil approved these changes Jul 21, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py Resolved

hkc-8010 requested review from bugraoz93, choo121600, henry3260, jason810496, pierrejeambrun, rawwar and shubhamraj-git as code owners July 22, 2026 03:51

hkc-8010 force-pushed the fix/dagrun-note-runtime-helper branch from 7f961c1 to 9b948ff Compare July 23, 2026 06:57

pierrejeambrun reviewed Jul 29, 2026

View reviewed changes

Comment thread task-sdk/src/airflow/sdk/execution_time/schema/versions/v2026_06_16.py Resolved

Comment thread airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py Resolved

hkc-8010 force-pushed the fix/dagrun-note-runtime-helper branch from dcc6b33 to f88aa4d Compare August 2, 2026 18:42

hkc-8010 requested a review from pierrejeambrun August 2, 2026 18:45

pierrejeambrun approved these changes Aug 20, 2026

View reviewed changes

hkc-8010 force-pushed the fix/dagrun-note-runtime-helper branch from f88aa4d to 58b99f7 Compare August 23, 2026 08:16

hkc-8010 force-pushed the fix/dagrun-note-runtime-helper branch from 58b99f7 to 765c589 Compare September 10, 2026 07:20

jason810496 reviewed Sep 10, 2026

View reviewed changes

hkc-8010 requested a review from jason810496 September 14, 2026 02:38

hkc-8010 force-pushed the fix/dagrun-note-runtime-helper branch from 01ebfe1 to 93bf5c0 Compare September 24, 2026 14:05

hkc-8010 and others added 6 commits September 24, 2026 21:58

Mark DagRun note updates as worker-only

d2a8729

The triggerer message-type guard needs to record worker-only runtime operations explicitly. Dag run note updates require a task execution context and should not become part of the triggerer supervisor protocol.

Preserve runtime DagRun notes on null update payloads

0a48d59

Address review feedback on runtime DagRun note updates

890658f

Skip the supervisor round-trip when the note is None, log when a runtime write replaces an attributed note, and rename the attribution test to say what it proves.

hkc-8010 force-pushed the fix/dagrun-note-runtime-helper branch from 93bf5c0 to 890658f Compare September 24, 2026 16:31

pierrejeambrun approved these changes Sep 30, 2026

View reviewed changes

pierrejeambrun merged commit 0b6c385 into apache:main Sep 30, 2026
108 checks passed

This was referenced Sep 30, 2026

Add task-handler parse messages and their parse-channel union #73973

Draft

Go SDK: answer TaskHandlerParseRequest with registered handlers #73975

Draft

vatsrahul1001 mentioned this pull request Oct 28, 2026

Status of testing of Apache Airflow 3.4.0rc3 #75865

Closed

alexcpn mentioned this pull request Nov 5, 2026

Ontology layer: typed relations, playbooks, guardrails, and catalogify find alexcpn/catalogify#22

Closed

Navigation Menu

Repository navigation

Add runtime DagRun note updates - #69403

Add runtime DagRun note updates#69403
pierrejeambrun merged 6 commits into
apache:mainfrom
hkc-8010:fix/dagrun-note-runtime-helper

hkc-8010 commented Jul 5, 2026

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Uh oh!

vatsrahul1001 commented Sep 10, 2026

Uh oh!

hkc-8010 commented Sep 10, 2026

Uh oh!

jason810496 left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

hkc-8010 commented Jul 5, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

vatsrahul1001 commented Sep 10, 2026

Uh oh!

hkc-8010 commented Sep 10, 2026

Uh oh!

jason810496 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pierrejeambrun left a comment •

edited

Loading

jason810496 left a comment •

edited

Loading