Skip to content

Allow Human-in-the-loop tasks to be answered while still running - #75413

Merged
kaxil merged 3 commits into
apache:mainfrom
astronomer:hitl-running-task-response
Nov 5, 2026
Merged

kaxil merged 3 commits into
apache:mainfrom
astronomer:hitl-running-task-response

Conversation

@kaxil

@kaxil kaxil commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

A Human-in-the-loop task can write its request from a running task, but it cannot read the answer: the task supervisor rejects GetHITLDetailResponse with 400 Unhandled request. This PR lets a running task read its own response, and makes the response endpoint, the Dags list and the UI treat a task that is still running while it checks for the answer as waiting. An operator can then pick up a quick answer before it parks in awaiting_input, without a trip through the scheduler. The provider side (response_poll_window on the HITL operators) is the follow-up PR stacked on this one.

Design rationale

The supervisor serves GetHITLDetailResponse through handle_get_hitl_detail_response in request_handlers.py, and the triggerer now uses the same handler and the same HITLDetailResponseResult instead of its own copy. Core requires Task SDK >=1.5.0, which ships the class; a backport to 3.4 would have to keep the triggerer's copy. The Execution API route is already scoped to ti:self, so a task can only read its own response.

When does a running task count as waiting? Every attempt gets a new task instance id, so a running task's request is always the one it wrote, and a retry can't be mistaken for the attempt that asked. It waits on its first run (next_method unset), or on a resumed run that asked again after the server resumed it, meaning the request's created_at is at or after scheduled_dttm, which every resume path stamps. A run resumed after a timeout without defaults isn't waiting, so #75645's refusal still holds for it. Both timestamps come from the server; the worker's clock never enters into it. HITLDetail.is_awaited_by() is the Python form and awaiting_response_clause() the SQL one, used by the Dags list "needs review" filter and its pending actions.

The response endpoint accepts an answer while the task waits by that rule and returns 409 otherwise, as #75645 does for a task no longer waiting. For a running task it only stores the answer: the task reads it itself, and if it parks first, the park transition resumes it with the answer. Only a parked task gets resumed by the endpoint.

Why the new hitl_detail.idempotency_key column? The Task SDK client retries POST /hitlDetails/{ti_id}, and the endpoint rewrites the request and clears any stored response. Once a running task can be answered, a reviewer could answer the request the first attempt stored, get a 200, and lose the answer to the retry. The client now sends an Idempotency-Key header per request (its retries reuse it), the endpoint stores it, and a POST that repeats it gets the stored request back unchanged. Content can't stand in for the key, since a resumed run may legitimately ask the same question again. I put it in a header rather than the request body so the request model, the supervisor schema and the Go and TypeScript SDKs stay as they are. An older client sends no key and behaves as today.

In the UI, the Required Actions list, the needs-review button and the tab count request and count running rows alongside deferred and awaiting_input. The UI has no next_method, so while a run resumed after a timeout is failing (a few seconds) it can still show the form; the server refuses that answer.

Required Actions form enabled while the task is still running

HITLDetailResponseResult is a new supervisor body, so schema.json and the vendored Go and TypeScript copies are regenerated. It's listed in v2027_02_28.py, the in-progress version.

Known issues

  • Only the latest request's key is stored. If a first attempt reaches the database after its own retry succeeded, the answer came in and the task asked again, it still overwrites the newer request. With the client's 5-second timeout, one request would have to sit unprocessed for minutes.
  • For a resumed run, created_at comes from the API server and scheduled_dttm from whichever process resumed the task (the scheduler, for a timeout), so the rule still compares two server clocks. The two writes are a full resume and re-ask apart.

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@bbovenzi

Copy link
Copy Markdown
Contributor

So the task state won't always be the orange "awaiting input"? We'll need to make sure the UX shows a user that there is a pending action. Should I help on that front as a follow up PR?

kaxil added 3 commits November 4, 2026 17:03
A running task could write its Human-in-the-loop request but not read the
answer: the task supervisor served CreateHITLDetailPayload and rejected
GetHITLDetailResponse as an unhandled request, since only the triggerer
needed it. Serve it from the task supervisor too, with a HITLDetailResponseResult
reply type in the Task SDK comms. The Execution API route is already scoped
to ti:self, so a task can only read its own response.

The UI and the pending-actions queries treated a HITL task as waiting for a
response only while it was deferred or awaiting_input. A task that checks
for its answer before parking is running at that point, so count it as
waiting too, but only while the attempt that wrote the request runs it for
the first time: an attempt that started after the request was written has
not asked yet, and a resumed run (for example a timeout without defaults)
has stopped waiting. The response endpoint refuses answers to a running
task in those two cases, which it used to accept and then lose.
…ker's clock

A running task instance's unanswered request is always its own, since
every attempt gets a new task instance id. So a running task waits on
its first run, or on a resumed run that asked again after the server
resumed it: compare created_at with scheduled_dttm, both stamped by the
server, instead of with the worker's start_date. That keeps refusing a
run resumed after a timeout, and accepts an operator that asks again
from a resumed run. The response endpoint resumes only a parked task;
a running one reads the answer itself.

The supervisor and the triggerer share one handler and one
HITLDetailResponseResult, since core requires the Task SDK that ships
it. List the new supervisor body under 2027-02-28, the in-progress
version, since 2026-10-30 ships in 3.4.
The Task SDK client retries POST /hitlDetails/{ti_id}, and the endpoint
rewrites the request and clears any stored response. Now that a running
task can be answered, a reviewer could answer the request the first
attempt stored, get a 200, and then lose the answer to the client's
retry. The client now sends an Idempotency-Key per request, reused by
its retries; the endpoint stores it and returns the stored request
unchanged when a POST repeats it.
@kaxil
kaxil force-pushed the hitl-running-task-response branch from 8d50460 to f1565a4 Compare November 4, 2026 18:41
@kaxil

kaxil commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

@bbovenzi Yeah, during response_poll_window it stays running because it's still holding the worker and pool slot, and pools count running. The needs-review button and Required Actions list already pick it up with this PR, but in the grid it just looks like a running task. A follow-up would be great, thanks. Maybe a "waiting for input" marker on a running TI with a pending request, linking to the form? But feel free to make it however you'd like.

@kaxil
kaxil marked this pull request as ready for review November 4, 2026 19:25

@bbovenzi bbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UI and API lgtm

@kaxil
kaxil merged commit 1f4bc05 into apache:main Nov 5, 2026
153 of 181 checks passed
@kaxil
kaxil deleted the hitl-running-task-response branch November 5, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:DAG-processing area:go-sdk area:task-sdk area:ts-sdk area:UI Related to UI/UX. For Frontend Developers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants