Skip to content

Avoid database queries while serializing Dag details and pools - #74295

Closed
rodrigopimentel wants to merge 1 commit into
apache:mainfrom
rodrigopimentel:feature/api-serialization-off-loop
Closed

rodrigopimentel wants to merge 1 commit into
apache:mainfrom
rodrigopimentel:feature/api-serialization-off-loop

Conversation

@rodrigopimentel

Copy link
Copy Markdown

FastAPI runs response serialization on the event loop, even for sync routes: serialize_response validates in the threadpool and then calls the serializer directly. Two public API responses queried the metadata database at that point. DAGDetailsResponse.latest_dag_version was a computed field calling DagVersion.get_latest_version on a new session, and PoolCollectionResponse.pools was an Iterable, which pydantic validates lazily, so the slot queries of every pool ran inside dump_json. When the SQLAlchemy pool is exhausted, the loop blocks on the checkout while the connections it waits for are returned by request teardowns that the loop schedules, so the API server stops serving, Execution API heartbeats and health probes included, until the pool timeout expires. Opening the Dag list, which requests /details for every row, was enough to trigger it in production; the issue has the details and a reproduction.

get_dag_details now loads the latest version through the request session, as it already does for is_favorite and active_runs_count, and latest_dag_version becomes a regular field. pools becomes a list, validated when the route builds the response, inside the threadpool. The OpenAPI spec, the airflowctl datamodels and the UI types are regenerated; latest_dag_version is no longer readOnly and the response shape is unchanged.

Checks run locally: the new tests with the existing TestDagDetails and TestGetPools in Breeze (45 passed; without the source change the two new tests fail with 6 and 1 queries during serialization), mypy on the changed files in Breeze, and the prek pre-commit and manual stages. The suites listed by breeze verify were not run locally and are left to CI.

closes: #74239


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

FastAPI serializes responses on the event loop, so a database query made
during serialization blocks the whole API server while it waits for a pool
connection. Once the pool is exhausted, the connections it waits for are
returned by request teardowns that also need the loop, and the server stops
answering, Execution API heartbeats and health probes included, until the
pool timeout expires.
@rodrigopimentel

Copy link
Copy Markdown
Author

Closing in favor of #74247, which fixed #74239 with the same approach

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API server event loop blocks on database queries while serializing get_dag_details and get_pools responses

1 participant