Repository navigation
Add an LLM task loop example that stops on a dollar or token budget - #75516
Merged
Merged
Conversation
The Common AI example Dag example_agent_loop_budget has an agent write a status page update and a second model score it, repeating both in a task loop until the score passes or the loop has spent a dollar or token budget. A terminal tally task sums each pass's usage XCom into loop.previous, and the gate returns a LoopDecision whose reason names the ending. A model with no price data turns the dollar cap off for the rest of the loop instead of counting as zero, so the token cap always applies. The new use-case page documents the four endings, the one-pass overshoot, estimated versus billed cost, unpriced models and retries. The core loops page links to it and states that an in-loop xcom_pull returns the current iteration's value.
Move the judge's system prompt to a module-level JUDGE_INSTRUCTIONS constant, next to INCIDENT_NOTES, so the numbered rules reach the model as a list instead of one run-on line.
pankajkoti
approved these changes
Oct 23, 2026
pankajkoti
left a comment
Member
There was a problem hiding this comment.
Thanks for the thorough write-up and for running every ending under a real scheduler. The screenshots made the behavior easy to follow. I have two inline comments and one note about a file outside the diff. All of them are questions or suggestions, so I'm happy to be told I misread something.
Outside the diff: providers/common/ai/docs/index.rst:38 says "ten jobs a data team already has" for the use cases. With this page there are now eleven. Would you like to change the number or drop it?
With max_iterations=5, the per-task usage limits capped the loop at 150,000 tokens and $3.50, below its $5.00 and 500,000-token budget, so neither budget stop could fire. Raise max_iterations to 20 and recapture the converged and max_iterations runs the page shows. Quote the provider's own log lines for a tripped per-task limit instead of a line printed by the example event listener, and drop the use-case count from the docs index so it does not go stale.
Member
Author
|
Thanks for the review, @pankajkoti. I dropped the number from |
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Common AI example Dag and a use-case page for a common ask: have an LLM keep improving something until a judge says it is good enough, without the loop as a whole spending more than a set amount.
example_agent_loop_budgethas@task.agentwrite a customer status page update and@task.llmscore it against a style guide. A task loop repeats both until the score reaches 0.9, or until the loop has spent $5.00 or 500,000 tokens. It needs Airflow 3.4 for task loops and Common AI 1.1.0, the first release in which@task.llmpushes theusageXCom.Nothing new is added to the API. The pattern uses what already exists: the loop's
untilreturning aLoopDecision, theusageXCom every Common AI operator pushes, and per-taskusage_limits. A terminaltallytask pulls the current iteration'susageXComs and adds them to the totals inloop.previous. The gate checks both caps after every pass and returns a reason that names how the loop ended.Design rationale
An unknown cost is never counted as zero. pydantic-ai reports
cost=Nonefor a model genai-prices can't price: a private fine-tune, a model newer than the price data, or thetestmodel. The first such call sets the loop's dollar total toNonefor the rest of the loop and logs a warning, and the gate's reasons saycost unknown. From then on only the token cap can stop the loop. One unpriced model is enough to switch the dollar cap off, so the page tells authors to set a token cap even when they budget in dollars.The judge uses
serialize_output=True. Without ittallyreceives aScoreinstance. That works under the scheduler, but underairflow dags testthe task fails withScore was not found in allow list for deserialization imports, becausedag.test()never runs the worker-side registration ofoutput_typeclasses. The use-case pages tell readers to runairflow dags test, so the example passes a dict. Fixingdags testitself is a separate core change.loops.rstgains one sentence: ati.xcom_pullof another task in the same loop returns that task's value from the current iteration. The example relies on it, and the page did not say what an in-loop pull returns.Runs
Every row ran the final example under a real scheduler with LocalExecutor. Only the caps, threshold and models changed between runs: Claude Sonnet 4.6 where a cost is shown, pydantic-ai's unpriced
testmodel where it sayscost unknown.passed: score 0.9; $0.023475 of $5.00, 4,757 of 500,000 tokensdollar budget spent: score 0.9; $0.016362 of $0.01, 3,226 of 500,000 tokenstoken budget spent: score 0.0; cost unknown, 745 of 600 tokenstoken budget spent: score 0.0; cost unknown, 2,063 of 2,000 tokensmax_iterationsreachedscore 0.5; $0.048030 of $5.00, 8,882 of 500,000 tokensLoopMaxIterationsExceededThese runs used
max_iterations=5. The example now ships 20: at 5, the per-task usage limits capped the loop at 150,000 tokens and $3.50, below its budget, so neither budget stop could fire. Re-run at 20 underairflow dags test, the converged case passed on pass 11 (passed: score 0.9; $0.096552 of $5.00, 18,624 of 500,000 tokens), and thetest-model case failed the last gate after 20 passes (score 0.0; cost unknown, 4,995 of 500,000 tokens). The screenshots below are from the runs at 5.In every iteration of every run,
tally's token total equals the running sum of that iteration's ownusageXComs, which confirms the in-loop pull.In the UI
The loop's outcome and the gate's reason, for a loop that converged:
Stopped on the dollar budget:
Stopped on the token budget, with both models unpriced:
Stopped on the token budget with a priced writer and an unpriced judge:
Reached
max_iterationswithout passing:Each gate shows its own decision, here the first pass of the converged run:
Known issues
usage_limitsbound that pass, give or take one model response.usageXCom holds the last try only, so the spend of an earlier try of a retried body task is missing from the totals. A forced retry spent 127 tokens on each try and the loop counted 127.{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.