[henry3260]

related: #76683

Replaces the prek hook from the first revision with a unit test, per the review question — the hook, its allowlist manager, its 33 tests and its .pre-commit-config.yaml entry are gone.

Why

A response field annotated with Iterable[...] is validated by pydantic lazily, while FastAPI serializes the response on the event loop. Each item's attributes are resolved there, on the sync engine: a lazy relationship or a selectin post-load becomes a query that blocks the whole API server once the connection pool is busy.

PoolCollectionResponse.pools was fixed by hand in #74247, which left an explanatory comment in the datamodel, and the dags and deadlines fields in #76525. Eight fields are still handed something unresolved by their route, and nothing stops a new one from being added — nothing about the annotation hints at the cost. list[...] validates on construction, inside the route, and renders to the same type: array in the OpenAPI schema, so the two read as interchangeable.

What

The generated/ + strict-xfail shape proposed for the same class of problem in #76424, rather than a static check.

airflow-core/tests/unit/api_fastapi/test_lazy_response_fields.py

• Imports every module under airflow.api_fastapi and collects the pydantic models from each module's namespace, not from pydantic.BaseModel.__subclasses__(). create_model attributes a model to the module that called it, so make_partial_model's output — and any model defined in another test module — shows up as a subclass and would make the result depend on what else the session had imported. Models nested in another model's body are picked up too.
• Reads each field's resolved annotation, so a bare Iterable, a TypeAliasType alias, Iterable[X] | None, Annotated[Iterable[X], Field(...)] and an inherited field all match. Only Iterable and Generator are actually validated lazily; Iterator / AsyncIterable / AsyncIterator raise PydanticSchemaGenerationError at class definition, and are listed to keep the rule one thing to remember.
• Rebuilds incomplete models first. An annotation naming something defined later in its own module stays an unparsed ForwardRef that get_origin cannot see through — AssetResponse.aliases is one — so a lazy field behind one would go unnoticed. A separate test fails if any model still cannot be resolved, so the check cannot quietly become a no-op.
• Skips main, gunicorn_app and gunicorn_config. None defines a model; importing main builds a whole app and sets _AIRFLOW_PROCESS_CONTEXT=server for the rest of the pytest process, and the gunicorn modules import an optional extra at module level.
• Guards every response model, not only *CollectionResponse: two of the existing fields are on plain response models in the execution API. A model defined inside a function, or one living outside the package, is out of reach, and the docstring says so.
• Covers the predicate with 23 parametrized cases and the two discovery paths with their own tests, and keeps a floor on the number of discovered models so a moved package cannot make the sweep pass vacuously.

The existing fields, split by what their route passes

Declaring these fields Iterable[...] was deliberate in #57878, to spare the routes a list() or .all() that SQLA2's typing had made them add. Whether that costs anything is decided at the route, which the annotation cannot show, so the twenty existing fields are split by reading every construction site.

The split is by what reaches the event loop, not by whether the route built a list. Pydantic validates each item during serialization either way, so an ORM row the response then lazy-loads from is as costly as passing the query itself — DAGRunCollectionResponse.dag_runs is exactly that case, through one of its four construction sites.

generated/known_lazy_response_fields.txt — 8 pending, tracked for migration in #76683: asset_aliases, backfills, connections, dag_runs, dag_versions, dag_warnings, deadline_alerts, variables. One <file>::<Class>.<field> per line, the same shape as known_api_server_threadpool_offloads.txt in #76424. The qualified name disambiguates the class names the core and execution APIs share, and no entry carries a line number, so an edit above one does not invalidate the list. Each line is a strict xfail, so every pending field is visible in the test report, and migrating one turns into an XPASS(strict) failure naming the exact line to delete. The list can only shrink. generated/* is in .rat-excludes, so the file carries no licence header.

LAZY_FIELDS_WITHOUT_SYNC_QUERY — 12 exempt. The items arrive already resolved, so validating them on the loop queries nothing. Eight routes build the response objects themselves — task_coordinate_responses and serialize_asset_events both return a list of responses and prefetch what building them needs. import_errors passes ORM rows, but the response reads only their own columns; jobs passes ORM rows too, and joinedload(Job.dag_model) covers the one relationship it reads through dag_display_name's AliasPath. TaskBreadcrumbsResponse.breadcrumbs is a generator, but over a result its route already awaited on the async engine, and HITLDetailResponse.chosen_options is a JSON column's value.

That set is closed: a new field has to be list[...], because a route that resolves its items today is one refactor away from not doing so. A test fails if an exempt entry stops being declared lazily, so a dead entry cannot sit there unnoticed. The annotation still permits the hazard in all twelve; no test can see that from the datamodel, which is the limit of this approach.

Verification

29 passed, 8 xfailed for the file, and 5775 passed, 30 skipped for the whole airflow-core/tests/unit/api_fastapi suite in a single process. The ratchet was exercised in every direction: a new lazy field (plain, bare and aliased) fails the sweep, migrating a pending one fails as a strict XPASS, and converting an exempt one to list[...] fails the staleness test. Rebasing onto main mid-review was itself a live test — #76525 had merged, so dags and deadlines turned into XPASS failures and their lines came out of the list.

The exempt/pending split was checked twice: once by reading every construction site, and once independently, which is how dag_runs was caught. The second pass also ran the route tests with a probe recording SQL issued during response serialization; the only statements it saw came from the bulk clearDagRuns route, the dag_runs site in question.

_read_known and _build_xfail_params are verbatim copies of #76424's helpers. Whichever of the two merges second should move them into devel-common/src/tests_common/test_utils/.

generated/README.md says the files there are hook-generated and are not to be edited by hand, which is not true of a ratchet like this one or of #76424's two lists. Worth settling in one place for all of them, rather than in either PR.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Claude Opus 5)

Generated-by: Claude Code (Claude Opus 5) following the guidelines

🤖 Generated with Claude Code

[henry3260]

cc @jason810496 PTAL

[jason810496]

Nice, thanks for covering up the regression.

High level question: Would it be possible to make this a unit test instead of prek hook?

Since we’re targeting to shrink the static check as much as possible, I’d like to double check the feasibility before adding a hook.

Thanks.

[henry3260]

Nice, thanks for covering up the regression.

High level question: Would it be possible to make this a unit test instead of prek hook?

Since we’re targeting to shrink the static check as much as possible, I’d like to double check the feasibility before adding a hook.

Thanks.

That's a good idea, changing to use unit test way

[jason810496]

IIRC, some of those fields in Pydantic model are intentionally kept as iterable instead of list - #57878.

Would it be possible to make those only involve SQLA query in the known exception rather than adding all of them as known invalidation?

Thanks.

[jason810496]

It seems like wrong issue id?

[henry3260]

IIRC, some of those fields in Pydantic model are intentionally kept as iterable instead of list - #57878.

Would it be possible to make those only involve SQLA query in the known exception rather than adding all of them as known invalidation?

Thanks.

Thanks for you were info. I measured it, and only DAGRunCollectionResponse.dag_runs actually queries on the event loop today, through bulk clearDagRuns.
So We'd fix that one, convert the six fields #57878 never touched (deadline_alerts, ClearTaskInstanceCollectionResponse.task_instances, dag_bundles, dag_bundle_files, chosen_options, breadcrumbs), and leave your thirteen as a closed exemption set so no new one can be added. The generated/ list and #76683 would then go.

Does that work? Or we should juxt fix DAGRunCollectionResponse.dag_runs