perf: use load_only() in get_dag_runs eager loading to reduce data fetched per task instance

[LakshmiSravyaVedantham]

Summary

Related #62025

The get_dag_runs REST API endpoint experiences slow response times on large Airflow deployments.

Root Cause

eager_load_dag_run_for_validation() used selectinload on DagRun.task_instances and DagRun.task_instances_histories without restricting which columns were fetched. This caused SQLAlchemy to load all heavyweight TaskInstance columns — including executor_config (a pickled binary blob), hostname, rendered_map_index, context_carrier, and many others — for every task instance across every DAG run in the result page.

In practice, only dag_version_id (the foreign key) is needed to traverse the TaskInstance.dag_version association proxy used by the DagRun.dag_versions property, which drives DAGRunResponse.dag_versions serialization.

Fix

Add load_only(TaskInstance.dag_version_id) and load_only(TaskInstanceHistory.dag_version_id) to the respective selectinload chains. SQLAlchemy will then issue a SELECT for task instances that fetches only the identity columns and dag_version_id, rather than the full row. The subsequent .joinedload(TaskInstance.dag_version).joinedload(DagVersion.bundle) chain continues to work correctly.

# Before
selectinload(DagRun.task_instances)
    .joinedload(TaskInstance.dag_version)
    .joinedload(DagVersion.bundle),

# After
selectinload(DagRun.task_instances)
    .load_only(TaskInstance.dag_version_id)
    .joinedload(TaskInstance.dag_version)
    .joinedload(DagVersion.bundle),

The same pattern is applied to task_instances_histories.

Impact

On deployments with many task instances per DAG run, this change significantly reduces:

• The volume of data transferred from the database per request
• Memory pressure from deserializing large pickled executor_config values
• Response latency for GET /dags/{dag_id}/dagRuns and the batch POST /dags/~/dagRuns/list endpoints

Testing

• Existing tests in tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py continue to pass
• The dag_versions property continues to work correctly since load_only() restricts deferred columns, not the relationship traversal

References

• SQLAlchemy load_only docs
• SQLAlchemy loading strategies
• Fixes: API - Improve public get_dag_runs endpoint #62025

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[pierrejeambrun]

Thanks,

Looks like it's doing some nice improvement. That's not enough to close the original issue, we need further improvement but that's a good start.

Lets see if the CI is happy.

Before

After

[pierrejeambrun]

Can you fix the static checks errors please (formatting and unused imports)

[pierrejeambrun]

I just pushed a commit to fix the static checks, it should look better 🤞

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[github-actions]

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-1-test