Repository navigation
Fix API client errors losing detail behind a reverse proxy - #75493
Merged
Merged
Conversation
airflowctl compared the error response content-type against the exact string "application/json". Any reverse proxy, ingress, or API gateway that returns "application/json; charset=utf-8" — or emits its own "application/problem+json" error — fell outside that comparison, so ServerResponseError was never constructed and the friendly "not found" messages in the dags and tasks commands were skipped. The bare httpx.HTTPStatusError raised instead is not handled by safe_call_command, so users saw a raw traceback. Airflow's own API server does not send a charset, which is why this only surfaces once something sits in front of it. Media types are case-insensitive and may carry parameters (RFC 9110), so the header has to be parsed rather than compared whole. Matching more responses also routes bodies into the error path that are valid JSON but not objects, which dict() rejected with a ValueError from that same unprotected path, and bodies that do not decode at all.
The Task SDK client carries the identical exact-string content-type comparison, so a task talking to the API server through a proxy that declares a charset loses the server's error detail and, because the raised exception is then a plain httpx.HTTPStatusError, no longer hits the status-specific handling in the supervisor and the task runner.
The airflowctl and Task SDK copies are the same three lines with different comments, and they already drifted in the commits that created them. A pointer in each makes the next person notice there are two.
Requested during review: the Task SDK half of the change ships in airflow-core, and both effects are visible to users running behind a proxy.
The Task SDK moved from httpx to httpx2 (apache#73116) after this branch was written, so the rebased helper and its tests referred to a module that is no longer imported there.
Since apache#73307 airflowctl catches the bare HTTPStatusError and prints one line instead of a traceback, so the newsfragment overstated it; the server's own message is still what gets lost.
The bare media type and the 404/500 axis repeat what test_error_parsing_with_unread_body already covers, and dict() of an object body worked before; none of them fail without the fix.
The previous comment described the dict() call the guard replaced, which is no longer in the code.
airflow-core's release notes ship only the Task SDK side; airflowctl's changelog is built from git log. The task runner never handled ServerResponseError, so it is no longer named.
rjgoyln
requested review from
amoghrajesh,
ashb,
bugraoz93,
dheerajturaga,
henry3260 and
potiuk
as code owners
October 23, 2026 15:09
The longer versions restated what the diff and the PR already explain, and the two _has_json_body copies now carry the same note.
henry3260
approved these changes
Oct 25, 2026
henry3260
left a comment
henry3260
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the fix! LGTM
henry3260
enabled auto-merge (squash)
October 25, 2026 17:05
Contributor
|
It’s weird that it’s taking so long to get merged. |
henry3260
disabled auto-merge
October 25, 2026 18:43
henry3260
enabled auto-merge (squash)
October 25, 2026 18:43
Contributor
Author
|
Thanks for the review and the branch update! |
Contributor
Backport successfully created: airflow-ctl/v0-1-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
potiuk
pushed a commit
that referenced
this pull request
Oct 25, 2026
…reverse proxy (#75493) (#75622) * Fix airflowctl tracebacks on JSON errors behind a reverse proxy airflowctl compared the error response content-type against the exact string "application/json". Any reverse proxy, ingress, or API gateway that returns "application/json; charset=utf-8" — or emits its own "application/problem+json" error — fell outside that comparison, so ServerResponseError was never constructed and the friendly "not found" messages in the dags and tasks commands were skipped. The bare httpx.HTTPStatusError raised instead is not handled by safe_call_command, so users saw a raw traceback. Airflow's own API server does not send a charset, which is why this only surfaces once something sits in front of it. Media types are case-insensitive and may carry parameters (RFC 9110), so the header has to be parsed rather than compared whole. Matching more responses also routes bodies into the error path that are valid JSON but not objects, which dict() rejected with a ValueError from that same unprotected path, and bodies that do not decode at all. * Move media type rationale from docstring into an inline comment * Apply the same JSON media type handling to the Task SDK client The Task SDK client carries the identical exact-string content-type comparison, so a task talking to the API server through a proxy that declares a charset loses the server's error detail and, because the raised exception is then a plain httpx.HTTPStatusError, no longer hits the status-specific handling in the supervisor and the task runner. * Cross-reference the twin JSON media type checks The airflowctl and Task SDK copies are the same three lines with different comments, and they already drifted in the commits that created them. A pointer in each makes the next person notice there are two. * Add newsfragment for the API client error handling fix Requested during review: the Task SDK half of the change ships in airflow-core, and both effects are visible to users running behind a proxy. * Use httpx2 in the Task SDK JSON media type check The Task SDK moved from httpx to httpx2 (#73116) after this branch was written, so the rebased helper and its tests referred to a module that is no longer imported there. * Describe the airflowctl symptom as it is on main now Since #73307 airflowctl catches the bare HTTPStatusError and prints one line instead of a traceback, so the newsfragment overstated it; the server's own message is still what gets lost. * Keep only the airflowctl test cases the fix changes The bare media type and the 404/500 axis repeat what test_error_parsing_with_unread_body already covers, and dict() of an object body worked before; none of them fail without the fix. * Explain the non-object log guard by what it protects The previous comment described the dict() call the guard replaced, which is no longer in the code. * Limit the newsfragment to the Task SDK fix airflow-core's release notes ship only the Task SDK side; airflowctl's changelog is built from git log. The task runner never handled ServerResponseError, so it is no longer named. * Rename the newsfragment to this PR's number * Shorten the media type comments The longer versions restated what the diff and the PR already explain, and the two _has_json_body copies now carry the same note. --------- (cherry picked from commit 352ca60) Co-authored-by: rjgoyln <151457491+rjgoyln@users.noreply.github.com> Co-authored-by: Henry Chen <henrychen@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #70936, which was approved and then closed by the open PR limit.
Summary
Both API clients parse an error body only when
content-typeis exactlyapplication/json. Behind a proxy that rewrites it toapplication/json; charset=utf-8, or one answering withapplication/problem+json:airflowctlshows only the HTTP status line, not the server's error message.HTTPStatusErrorinstead ofServerResponseError, so the status-specific handling in the supervisor and the client's operations (a 404 meaning a missing connection or variable, for example) stops matching.Airflow's own API server sends the bare media type, so this needs a proxy in between.
Change
+jsonsuffixes, in both clients.airflowctl, treat an undecodable JSON body as a plain HTTP error, and log a non-object JSON body instead of crashing on it.Since the approval on #70936 the branch is rebased onto the Task SDK's
httpx2migration (#73116), and the newsfragment covers only the Task SDK side, which is what ships in airflow-core.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines