Repository navigation
Avoid database queries while serializing Dag details and pools - #74295
Closed
rodrigopimentel wants to merge 1 commit into
Closed
rodrigopimentel wants to merge 1 commit into
rodrigopimentel wants to merge 1 commit into
Conversation
FastAPI serializes responses on the event loop, so a database query made during serialization blocks the whole API server while it waits for a pool connection. Once the pool is exhausted, the connections it waits for are returned by request teardowns that also need the loop, and the server stops answering, Execution API heartbeats and health probes included, until the pool timeout expires.
rodrigopimentel
requested review from
bbovenzi,
bugraoz93,
choo121600,
dheerajturaga,
ephraimbuddy,
guan404ming,
henry3260,
jason810496,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git and
vatsrahul1001
as code owners
October 5, 2026 19:07
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FastAPI runs response serialization on the event loop, even for sync routes:
serialize_responsevalidates in the threadpool and then calls the serializer directly. Two public API responses queried the metadata database at that point.DAGDetailsResponse.latest_dag_versionwas a computed field callingDagVersion.get_latest_versionon a new session, andPoolCollectionResponse.poolswas anIterable, which pydantic validates lazily, so the slot queries of every pool ran insidedump_json. When the SQLAlchemy pool is exhausted, the loop blocks on the checkout while the connections it waits for are returned by request teardowns that the loop schedules, so the API server stops serving, Execution API heartbeats and health probes included, until the pool timeout expires. Opening the Dag list, which requests/detailsfor every row, was enough to trigger it in production; the issue has the details and a reproduction.get_dag_detailsnow loads the latest version through the request session, as it already does foris_favoriteandactive_runs_count, andlatest_dag_versionbecomes a regular field.poolsbecomes alist, validated when the route builds the response, inside the threadpool. The OpenAPI spec, the airflowctl datamodels and the UI types are regenerated;latest_dag_versionis no longerreadOnlyand the response shape is unchanged.Checks run locally: the new tests with the existing
TestDagDetailsandTestGetPoolsin Breeze (45 passed; without the source change the two new tests fail with 6 and 1 queries during serialization), mypy on the changed files in Breeze, and theprekpre-commit and manual stages. The suites listed bybreeze verifywere not run locally and are left to CI.closes: #74239
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines