Repository navigation
Publish usage, run id and model name from LLMOperator's subclasses - #75208
Merged
pierrejeambrun merged 7 commits intoOct 29, 2026
Merged
Conversation
pierrejeambrun
marked this pull request as ready for review
October 20, 2026 14:21
pierrejeambrun
requested review from
Lee-W,
ashb,
gopidesupavan and
kaxil
as code owners
October 20, 2026 14:21
pierrejeambrun
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 20, 2026
pierrejeambrun
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 20, 2026
kaxil
reviewed
Oct 21, 2026
kaxil
approved these changes
Oct 22, 2026
pierrejeambrun
force-pushed
the
llm-subclasses-run-telemetry
branch
from
October 28, 2026 11:37
43ab8a6 to
8579d01
Compare
LLMBranchOperator, LLMSQLQueryOperator, LLMSchemaCompareOperator and LLMFileAnalysisOperator override execute() without calling super().execute(), so they never pushed the usage, run_id and model name XComs, never stamped the task-instance identity on their GenAI spans, and had no AI Model tab. Calling super() is not an option: it would also apply LLMOperator's decision record, confidence gate and review body, which these operators replace or don't have. The reporting now lives in _run_llm, the one place every execute() runs the agent, so the subclasses, and any future one, report the same metadata as LLMOperator.
The Scope bullet read as if agents traced through agent_framework_tracing were excluded from everything, when they still get the airflow.* identity attributes and only miss the XComs.
Both methods were named after running, which hid that _run_llm is the entry point every execute() calls and the other one only runs the agent under the usage budget.
The subclass tests passed even when _run_llm forwarded no run id or skipped the identity stamp, and the AI Model tab's operator list could miss a new LLMOperator subclass unnoticed. The tests now fail in each of those cases.
The SQL, branch, file-analysis and schema-compare pages stopped at the log summary, so a reader of those pages could not tell the operators also push usage, run_id and the model name. The XCom tests used an unspecced task instance, so any attribute the identity stamping read would silently pass.
pierrejeambrun
force-pushed
the
llm-subclasses-run-telemetry
branch
from
October 28, 2026 12:06
8579d01 to
37e7402
Compare
The anchor sits inside a bullet rather than before a section, so Sphinx has no title to use for a bare reference to it.
… anchor Every other operator page, llm.rst included, points at the observability page as a whole; an anchor inside a bullet was a one-off pattern.
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#74272 and #74355 made
LLMOperatorpublish its model name,usageandrun_idon XCom and stamp the task-instance identity on its GenAI spans. #74334 added the AI Model tab that reads those XComs.LLMBranchOperator,LLMSQLQueryOperator,LLMSchemaCompareOperatorandLLMFileAnalysisOperatorgot none of this, because they overrideexecute()without callingsuper().execute(). You couldn't see what a branch or SQL-generation task cost, or which model answered, without reading its logs. This was left as follow-up work in those PRs.Calling
super().execute()from the subclasses would not work.LLMOperator.execute()also writes the generic decision record and applies the confidence gate, theon_uncertainpolicy and its own review body.LLMBranchOperatorreplaces all of that with its own decision record. The SQL, schema-compare and file-analysis operators don't use any of it. Copying the reporting block into four overrides would mean four copies of the failure path and its guards to keep in sync.So the reporting now happens in
_run_llm, which every one of theseexecute()methods already calls to run the agent. The four subclasses report the same metadata asLLMOperatorwithout changing their own code, and so would any future subclass. The AI Model tab is now scoped to the four operators and their@task.llm_*decorators too.LLMBatchOperatoris not anLLMOperatorsubclass and is unchanged.The changes to existing subclass tests are mechanical. The agent mocks now allow
instrument, since identity stamping runs. Therun_synccall assertions now include theusage(and, where a task instance is present,run_id) the run is called with.I ran each operator and its
@task.llm_*decorator against a real model in a local Airflow. Every task pushedusage,run_idand the model name, and its AI Model tab shows them:LLMBranchOperatorLLMSQLQueryOperatorLLMSchemaCompareOperatorLLMFileAnalysisOperatorrelated: #74272, #74334, #74355
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines