Skip to content

Fix airflow db check reporting one fewer retry than it will make - #72708

Merged
potiuk merged 1 commit into
apache:mainfrom
Eason09053360:fix-db-check-remaining-retries-count
Sep 22, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
Eason09053360:fix-db-check-remaining-retries-count

Conversation

@Eason09053360

Copy link
Copy Markdown
Contributor

airflow db check --retry N logs a countdown between attempts that is off by one, so the last
message before the final attempt contradicts itself:

$ airflow db check --retry 3 --retry-delay 9
3 retries remain. Will retry in 9 seconds     <- was "2 retries remain"
2 retries remain. Will retry in 9 seconds     <- was "1 retries remain"
1 retries remain. Will retry in 9 seconds     <- was "0 retries remain. Will retry in 9 seconds"
<final attempt, then the OperationalError is re-raised>

The loop runs stop_after_attempt(1 + retries), and tenacity's before_sleep hook fires only when
another attempt will follow, with attempt_number being the attempt that has just failed. So after
attempt n there are (1 + retries) - n attempts left. The callback computed
retries - attempt_number, one lower, and the loop's + 1 was never mirrored in the message.

The visible consequence is the final line: 0 retries remain. Will retry in 9 seconds — it
announces that the command has given up at the exact moment it is about to try again.

Impact

This is the message people read while a container waits for its database. airflow db check is what
scripts/docker/entrypoint_prod.sh runs (via wait_for_airflow_db) before the rest of the
entrypoint proceeds, so the countdown is often the only feedback during a slow database start. A
countdown that reaches zero and keeps going makes it look like the wait loop is stuck rather than
progressing, which is exactly the wrong signal to send someone debugging a stalled deployment.

Changes

One line: remain = retries - retrystate.attempt_number + 1.

Tests

test_check_warns_about_the_retries_that_are_actually_left drives db check --retry 3 with the
database check patched to always fail, and asserts on the captured log via the caplog fixture:

assert "3 retries remain. Will retry in 9 seconds" in caplog
assert "2 retries remain. Will retry in 9 seconds" in caplog
assert "1 retries remain. Will retry in 9 seconds" in caplog
assert "0 retries remain. Will retry in 9 seconds" not in caplog

The final negative assertion pins the defect itself — the self-contradictory line must not appear.

Asserting on the captured log rather than on the arguments handed to a mocked logger also covers the
rendering: the module uses structlog, so this confirms the %d placeholders are actually
interpolated and not printed literally.

Reverting the one-line change and running the whole file fails this test; with the change, all 118
tests in the file pass. --retry 0 is unaffected: before_sleep never fires, so no countdown is
logged at all.

Notes for reviewers

The corrected countdown now ends at 1, so the message reads 1 retries remain. The awkward plural
is pre-existing wording that this change does not touch; happy to fix it here if preferred.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

The warning printed between attempts computed the remaining count as
retries minus the attempt that had just failed, one lower than the loop's own
stop_after_attempt(1 + retries) budget. The last message before the final
attempt therefore read "0 retries remain. Will retry in N seconds", telling
anyone watching a container wait for its database that the command had given
up at the moment it was still trying.
@Eason09053360
Eason09053360 force-pushed the fix-db-check-remaining-retries-count branch 2 times, most recently from 6ed9eae to e196ed6 Compare September 9, 2026 14:42
@fat-catTW

Copy link
Copy Markdown
Contributor

Thanks for the fix!

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — correct fix and a clear write-up. With stop_after_attempt(1 + retries), before_sleep sees the attempt that just failed, so (1 + retries) - attempt_number attempts remain — exactly what the + 1 gives. The new test fails on main (no "3 retries remain", and a "0 retries remain" line) and passes here. Approving.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants