Repository navigation
Fix airflow db check reporting one fewer retry than it will make - #72708
Conversation
The warning printed between attempts computed the remaining count as retries minus the attempt that had just failed, one lower than the loop's own stop_after_attempt(1 + retries) budget. The last message before the final attempt therefore read "0 retries remain. Will retry in N seconds", telling anyone watching a container wait for its database that the command had given up at the moment it was still trying.
6ed9eae to
e196ed6
Compare
|
Thanks for the fix! |
potiuk
left a comment
There was a problem hiding this comment.
Thanks — correct fix and a clear write-up. With stop_after_attempt(1 + retries), before_sleep sees the attempt that just failed, so (1 + retries) - attempt_number attempts remain — exactly what the + 1 gives. The new test fails on main (no "3 retries remain", and a "0 retries remain" line) and passes here. Approving.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
airflow db check --retry Nlogs a countdown between attempts that is off by one, so the lastmessage before the final attempt contradicts itself:
The loop runs
stop_after_attempt(1 + retries), and tenacity'sbefore_sleephook fires only whenanother attempt will follow, with
attempt_numberbeing the attempt that has just failed. So afterattempt n there are
(1 + retries) - nattempts left. The callback computedretries - attempt_number, one lower, and the loop's+ 1was never mirrored in the message.The visible consequence is the final line:
0 retries remain. Will retry in 9 seconds— itannounces that the command has given up at the exact moment it is about to try again.
Impact
This is the message people read while a container waits for its database.
airflow db checkis whatscripts/docker/entrypoint_prod.shruns (viawait_for_airflow_db) before the rest of theentrypoint proceeds, so the countdown is often the only feedback during a slow database start. A
countdown that reaches zero and keeps going makes it look like the wait loop is stuck rather than
progressing, which is exactly the wrong signal to send someone debugging a stalled deployment.
Changes
One line:
remain = retries - retrystate.attempt_number + 1.Tests
test_check_warns_about_the_retries_that_are_actually_leftdrivesdb check --retry 3with thedatabase check patched to always fail, and asserts on the captured log via the
caplogfixture:The final negative assertion pins the defect itself — the self-contradictory line must not appear.
Asserting on the captured log rather than on the arguments handed to a mocked logger also covers the
rendering: the module uses
structlog, so this confirms the%dplaceholders are actuallyinterpolated and not printed literally.
Reverting the one-line change and running the whole file fails this test; with the change, all 118
tests in the file pass.
--retry 0is unaffected:before_sleepnever fires, so no countdown islogged at all.
Notes for reviewers
The corrected countdown now ends at
1, so the message reads1 retries remain. The awkward pluralis pre-existing wording that this change does not touch; happy to fix it here if preferred.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines