Repository navigation
Keep get_dag_details and get_pools database queries off the API server event loop - #74247
Merged
henry3260 merged 2 commits intoOct 7, 2026
Conversation
schizophrenicmaniac
requested review from
bbovenzi,
bugraoz93,
choo121600,
dheerajturaga,
ephraimbuddy,
guan404ming,
henry3260,
jason810496,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
October 5, 2026 14:00
boring-cyborg
Bot
added
area:airflow-ctl
area:API
rjgoyln
approved these changes
Oct 6, 2026
rjgoyln
left a comment
rjgoyln
left a comment
Contributor
There was a problem hiding this comment.
LGTM! The static-checks failure looks unrelated.
henry3260
force-pushed
the
fix/74239-api-server-event-loop-blocks-on
branch
from
October 6, 2026 13:31
dac87e4 to
36d5760
Compare
Contributor
Backport failed to create: airflow-ctl/v0-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 9af3db0 airflow-ctl/v0-1-testThis should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
This was referenced Oct 7, 2026
1 task done
1 of 8 tasks
1 task done
1 task done
henry3260
added a commit
to henry3260/airflow
that referenced
this pull request
Nov 7, 2026
A response field annotated with Iterable is validated by pydantic lazily, while FastAPI serializes the response on the event loop. Each item's attributes are resolved there, on the sync engine: a lazy relationship or a selectin post-load becomes a query that blocks the whole API server once the connection pool is busy. PoolCollectionResponse.pools was fixed by hand in apache#74247, which left an explanatory comment in the datamodel, and the dags and deadlines fields in apache#76525. Twenty fields still have the shape, and nothing stops a new one from being added, because nothing about the annotation hints at the cost: list[...] validates on construction, inside the route, and renders to the same type: array in the OpenAPI schema, so the two read as interchangeable. The known fields are grandfathered as strict xfails rather than fixed here, so the list stays visible in the test report and can only shrink. The shape of the check follows the one proposed for the same class of problem in apache#76424. Incomplete models are rebuilt before their fields are read. An annotation naming something defined later in its own module stays an unparsed ForwardRef that get_origin cannot see through, as AssetResponse.aliases does, so a lazy field behind one would go unnoticed.
henry3260
added a commit
to henry3260/airflow
that referenced
this pull request
Nov 7, 2026
A response field annotated with Iterable is validated by pydantic lazily, while FastAPI serializes the response on the event loop. Each item's attributes are resolved there, on the sync engine: a lazy relationship or a selectin post-load becomes a query that blocks the whole API server once the connection pool is busy. PoolCollectionResponse.pools was fixed by hand in apache#74247, which left an explanatory comment in the datamodel, and the dags and deadlines fields in apache#76525. Twenty fields still have the shape, and nothing stops a new one from being added, because nothing about the annotation hints at the cost: list[...] validates on construction, inside the route, and renders to the same type: array in the OpenAPI schema, so the two read as interchangeable. The known fields are grandfathered as strict xfails rather than fixed here, so the list stays visible in the test report and can only shrink. The shape of the check follows the one proposed for the same class of problem in apache#76424. Incomplete models are rebuilt before their fields are read. An annotation naming something defined later in its own module stays an unparsed ForwardRef that get_origin cannot see through, as AssetResponse.aliases does, so a lazy field behind one would go unnoticed.
henry3260
added a commit
to henry3260/airflow
that referenced
this pull request
Nov 7, 2026
A response field annotated with Iterable is validated by pydantic lazily, while FastAPI serializes the response on the event loop. Each item's attributes are resolved there, on the sync engine: a lazy relationship or a selectin post-load becomes a query that blocks the whole API server once the connection pool is busy. PoolCollectionResponse.pools was fixed by hand in apache#74247, which left an explanatory comment in the datamodel, and the dags and deadlines fields in apache#76525. Twenty fields still have the shape, and nothing stops a new one from being added, because nothing about the annotation hints at the cost: list[...] validates on construction, inside the route, and renders to the same type: array in the OpenAPI schema, so the two read as interchangeable. The known fields are grandfathered as strict xfails rather than fixed here, so the list stays visible in the test report and can only shrink. The shape of the check follows the one proposed for the same class of problem in apache#76424. Incomplete models are rebuilt before their fields are read. An annotation naming something defined later in its own module stays an unparsed ForwardRef that get_origin cannot see through, as AssetResponse.aliases does, so a lazy field behind one would go unnoticed.
1 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FastAPI validates a sync route's response in the threadpool but serializes it on the event loop. Two public API responses were still hitting the database during serialization, so when the SQLAlchemy pool was exhausted the whole API server stalled on the pool checkout until
pool_timeoutexpired.DAGDetailsResponse.latest_dag_versionwas a computed field that opened its own session to load the latest DagVersion.get_dag_detailsnow loads it through the request session and sets it on the model, likeis_favoriteandactive_runs_count, andlatest_dag_versionis a regular field. This also drops the second connection each/detailsrequest was taking.PoolCollectionResponse.poolswasIterable[PoolResponse], which pydantic validates lazily during serialization, so the six slot queries per pool ran insidedump_json. It is nowlist[PoolResponse], so they run when the route builds the response in the threadpool.In the OpenAPI schema
latest_dag_versionstays required and nullable, but it is no longer markedreadOnly. I regenerated the spec, the UI client and the airflow-ctl datamodels.The new tests record SQL statements executed on the event loop thread while serving
/dags/{dag_id}/detailsand/pools. Without the change they catch 1 and 24 statements respectively.closes: #74239
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.