Skip to content

Add an LLM task loop example that stops on a dollar or token budget - #75516

Merged
kaxil merged 3 commits into
apache:mainfrom
astronomer:loop-budget-example
Oct 23, 2026
Merged

kaxil merged 3 commits into
apache:mainfrom
astronomer:loop-budget-example

Conversation

@kaxil

@kaxil kaxil commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Adds a Common AI example Dag and a use-case page for a common ask: have an LLM keep improving something until a judge says it is good enough, without the loop as a whole spending more than a set amount.

example_agent_loop_budget has @task.agent write a customer status page update and @task.llm score it against a style guide. A task loop repeats both until the score reaches 0.9, or until the loop has spent $5.00 or 500,000 tokens. It needs Airflow 3.4 for task loops and Common AI 1.1.0, the first release in which @task.llm pushes the usage XCom.

Nothing new is added to the API. The pattern uses what already exists: the loop's until returning a LoopDecision, the usage XCom every Common AI operator pushes, and per-task usage_limits. A terminal tally task pulls the current iteration's usage XComs and adds them to the totals in loop.previous. The gate checks both caps after every pass and returns a reason that names how the loop ended.

Design rationale

An unknown cost is never counted as zero. pydantic-ai reports cost=None for a model genai-prices can't price: a private fine-tune, a model newer than the price data, or the test model. The first such call sets the loop's dollar total to None for the rest of the loop and logs a warning, and the gate's reasons say cost unknown. From then on only the token cap can stop the loop. One unpriced model is enough to switch the dollar cap off, so the page tells authors to set a token cap even when they budget in dollars.

The judge uses serialize_output=True. Without it tally receives a Score instance. That works under the scheduler, but under airflow dags test the task fails with Score was not found in allow list for deserialization imports, because dag.test() never runs the worker-side registration of output_type classes. The use-case pages tell readers to run airflow dags test, so the example passes a dict. Fixing dags test itself is a separate core change.

loops.rst gains one sentence: a ti.xcom_pull of another task in the same loop returns that task's value from the current iteration. The example relies on it, and the page did not say what an in-loop pull returns.

Runs

Every row ran the final example under a real scheduler with LocalExecutor. Only the caps, threshold and models changed between runs: Claude Sonnet 4.6 where a cost is shown, pydantic-ai's unpriced test model where it says cost unknown.

Ending Passes Reason on the last gate Run
Converged 3 passed: score 0.9; $0.023475 of $5.00, 4,757 of 500,000 tokens success
Dollar budget, $0.01 cap 2 dollar budget spent: score 0.9; $0.016362 of $0.01, 3,226 of 500,000 tokens success
Token budget, 600-token cap, both models unpriced 3 token budget spent: score 0.0; cost unknown, 745 of 600 tokens success
Token budget, 2,000-token cap, priced writer and unpriced judge 3 token budget spent: score 0.0; cost unknown, 2,063 of 2,000 tokens success
max_iterations reached 5 score 0.5; $0.048030 of $5.00, 8,882 of 500,000 tokens last gate fails with LoopMaxIterationsExceeded

These runs used max_iterations=5. The example now ships 20: at 5, the per-task usage limits capped the loop at 150,000 tokens and $3.50, below its budget, so neither budget stop could fire. Re-run at 20 under airflow dags test, the converged case passed on pass 11 (passed: score 0.9; $0.096552 of $5.00, 18,624 of 500,000 tokens), and the test-model case failed the last gate after 20 passes (score 0.0; cost unknown, 4,995 of 500,000 tokens). The screenshots below are from the runs at 5.

In every iteration of every run, tally's token total equals the running sum of that iteration's own usage XComs, which confirms the in-loop pull.

In the UI

The loop's outcome and the gate's reason, for a loop that converged:

Loop converged at iteration 3

Stopped on the dollar budget:

Loop stopped on the dollar budget

Stopped on the token budget, with both models unpriced:

Loop stopped on the token budget with an unknown cost

Stopped on the token budget with a priced writer and an unpriced judge:

Loop stopped on the token budget with mixed pricing

Reached max_iterations without passing:

Loop failed at max_iterations

Each gate shows its own decision, here the first pass of the converged run:

Gate decision on iteration 1

Known issues

  • Either cap can be overshot by one pass: the pass that crosses it has already run when the gate checks. A 600-token cap stopped at 745 after a 250-token pass. Per-task usage_limits bound that pass, give or take one model response.
  • The usage XCom holds the last try only, so the spend of an earlier try of a retried body task is missing from the totals. A forced retry spent 127 tokens on each try and the loop counted 127.
  • The dollar total is the genai-prices estimate from list prices, not the provider's bill.

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

The Common AI example Dag example_agent_loop_budget has an agent write a
status page update and a second model score it, repeating both in a task
loop until the score passes or the loop has spent a dollar or token budget.
A terminal tally task sums each pass's usage XCom into loop.previous, and
the gate returns a LoopDecision whose reason names the ending.

A model with no price data turns the dollar cap off for the rest of the
loop instead of counting as zero, so the token cap always applies.

The new use-case page documents the four endings, the one-pass overshoot,
estimated versus billed cost, unpriced models and retries. The core loops
page links to it and states that an in-loop xcom_pull returns the current
iteration's value.
Move the judge's system prompt to a module-level JUDGE_INSTRUCTIONS
constant, next to INCIDENT_NOTES, so the numbered rules reach the model
as a list instead of one run-on line.
@kaxil
kaxil requested review from phanikumv and shahar1 October 23, 2026 19:36

@pankajkoti pankajkoti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough write-up and for running every ending under a real scheduler. The screenshots made the behavior easy to follow. I have two inline comments and one note about a file outside the diff. All of them are questions or suggestions, so I'm happy to be told I misread something.

Outside the diff: providers/common/ai/docs/index.rst:38 says "ten jobs a data team already has" for the use cases. With this page there are now eleven. Would you like to change the number or drop it?

With max_iterations=5, the per-task usage limits capped the loop at 150,000
tokens and $3.50, below its $5.00 and 500,000-token budget, so neither budget
stop could fire. Raise max_iterations to 20 and recapture the converged and
max_iterations runs the page shows.

Quote the provider's own log lines for a tripped per-task limit instead of a
line printed by the example event listener, and drop the use-case count from
the docs index so it does not go stale.
@kaxil

kaxil commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review, @pankajkoti. I dropped the number from index.rst, so it reads "jobs a data team already has" and won't go stale with the next use case. Both inline points are addressed in d484ede.

@kaxil
kaxil merged commit 57a4751 into apache:main Oct 23, 2026
74 checks passed
@kaxil
kaxil deleted the loop-budget-example branch October 23, 2026 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants