Repository navigation
Stabilize testcontainers provider tests on slow (ARM) CI runners - #68963
Merged
Merged
Conversation
The mongo provider tests boot a real mongo:8.0 container via testcontainers in the setup phase of the first test that uses the session fixture. That bring-up -- image pull, container start, readiness ping -- has to fit inside the per-test pytest setup-timeout (60s). On the GitHub-hosted ARM canary, where the image cache is cold and registry/disk throughput is lower, the pull alone can eat most of that budget, so setup times out and cascades into "ERROR at setup" across every TestMongoHook test. AMD does not hit this because it runs on every PR with warm caches; ARM only runs on the cron canary. Two complementary changes: - Pre-pull the testcontainers images (mongo:8.0 + ryuk) on the shared host daemon before the timed run, but only in CI and only when the owning provider's tests are actually in the selection -- so the slow cold-cache pull happens outside the per-test setup timeout. Locally it stays a no-op. - Give pytest setup/teardown timeouts a higher floor (180s) for container- backed fixtures, while keeping execution-timeout at the per-test value so genuinely hung tests are still caught quickly.
potiuk
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl and
vatsrahul1001
as code owners
June 25, 2026 06:54
bugraoz93
approved these changes
Jun 25, 2026
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
github-actions Bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jun 26, 2026
…runners (apache#68963) The mongo provider tests boot a real mongo:8.0 container via testcontainers in the setup phase of the first test that uses the session fixture. That bring-up -- image pull, container start, readiness ping -- has to fit inside the per-test pytest setup-timeout (60s). On the GitHub-hosted ARM canary, where the image cache is cold and registry/disk throughput is lower, the pull alone can eat most of that budget, so setup times out and cascades into "ERROR at setup" across every TestMongoHook test. AMD does not hit this because it runs on every PR with warm caches; ARM only runs on the cron canary. Two complementary changes: - Pre-pull the testcontainers images (mongo:8.0 + ryuk) on the shared host daemon before the timed run, but only in CI and only when the owning provider's tests are actually in the selection -- so the slow cold-cache pull happens outside the per-test setup timeout. Locally it stays a no-op. - Give pytest setup/teardown timeouts a higher floor (180s) for container- backed fixtures, while keeping execution-timeout at the per-test value so genuinely hung tests are still caught quickly. (cherry picked from commit 9b0fe28) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
aws-airflow-bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jun 26, 2026
…runners (apache#68963) The mongo provider tests boot a real mongo:8.0 container via testcontainers in the setup phase of the first test that uses the session fixture. That bring-up -- image pull, container start, readiness ping -- has to fit inside the per-test pytest setup-timeout (60s). On the GitHub-hosted ARM canary, where the image cache is cold and registry/disk throughput is lower, the pull alone can eat most of that budget, so setup times out and cascades into "ERROR at setup" across every TestMongoHook test. AMD does not hit this because it runs on every PR with warm caches; ARM only runs on the cron canary. Two complementary changes: - Pre-pull the testcontainers images (mongo:8.0 + ryuk) on the shared host daemon before the timed run, but only in CI and only when the owning provider's tests are actually in the selection -- so the slow cold-cache pull happens outside the per-test setup timeout. Locally it stays a no-op. - Give pytest setup/teardown timeouts a higher floor (180s) for container- backed fixtures, while keeping execution-timeout at the per-test value so genuinely hung tests are still caught quickly. (cherry picked from commit 9b0fe28) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
potiuk
added a commit
that referenced
this pull request
Jun 26, 2026
…runners (#68963) (#69054) The mongo provider tests boot a real mongo:8.0 container via testcontainers in the setup phase of the first test that uses the session fixture. That bring-up -- image pull, container start, readiness ping -- has to fit inside the per-test pytest setup-timeout (60s). On the GitHub-hosted ARM canary, where the image cache is cold and registry/disk throughput is lower, the pull alone can eat most of that budget, so setup times out and cascades into "ERROR at setup" across every TestMongoHook test. AMD does not hit this because it runs on every PR with warm caches; ARM only runs on the cron canary. Two complementary changes: - Pre-pull the testcontainers images (mongo:8.0 + ryuk) on the shared host daemon before the timed run, but only in CI and only when the owning provider's tests are actually in the selection -- so the slow cold-cache pull happens outside the per-test setup timeout. Locally it stays a no-op. - Give pytest setup/teardown timeouts a higher floor (180s) for container- backed fixtures, while keeping execution-timeout at the per-test value so genuinely hung tests are still caught quickly. (cherry picked from commit 9b0fe28) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
karenbraganz
pushed a commit
to karenbraganz/airflow
that referenced
this pull request
Jun 30, 2026
…che#68963) The mongo provider tests boot a real mongo:8.0 container via testcontainers in the setup phase of the first test that uses the session fixture. That bring-up -- image pull, container start, readiness ping -- has to fit inside the per-test pytest setup-timeout (60s). On the GitHub-hosted ARM canary, where the image cache is cold and registry/disk throughput is lower, the pull alone can eat most of that budget, so setup times out and cascades into "ERROR at setup" across every TestMongoHook test. AMD does not hit this because it runs on every PR with warm caches; ARM only runs on the cron canary. Two complementary changes: - Pre-pull the testcontainers images (mongo:8.0 + ryuk) on the shared host daemon before the timed run, but only in CI and only when the owning provider's tests are actually in the selection -- so the slow cold-cache pull happens outside the per-test setup timeout. Locally it stays a no-op. - Give pytest setup/teardown timeouts a higher floor (180s) for container- backed fixtures, while keeping execution-timeout at the per-test value so genuinely hung tests are still caught quickly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The mongo provider tests boot a real
mongo:8.0container via testcontainers (since #53773 dropped mongomock). The whole bring-up — image pull, container start, readiness ping — happens in the setup phase of the first test that uses the session fixture, and must fit inside the per-test pytest--setup-timeout(60s).On the GitHub-hosted ARM canary (
ubuntu-22.04-arm), where the Docker image cache is cold and registry/disk throughput is lower, themongo:8.0pull alone (~17–19s observed) eats much of that 60s budget, so setup times out → cascade ofERROR at setup of TestMongoHook.*(Failed: Timeout >60.0s). AMD doesn't hit it because it runs on every PR with warm caches; ARM only runs on the cron canary. (Same run, Py3.11 mongo passed while Py3.10 timed out — a timing/infra signature, not a code defect, and not emulation: ARM runners are native arm64 with a nativemongo:8.0image, no QEMU.)Changes
A. Pre-pull testcontainers images out of the timed path. Before a parallel provider run, pull the images on the shared host daemon — but only in CI, and only when the owning provider's tests are actually selected (e.g. skip when the run is
Providers[mysql,...]without mongo, or excludes it). Locally it's a no-op; testcontainers pulls on demand as before. Keyed by provider inTESTCONTAINERS_IMAGES_BY_PROVIDERso it scales to future testcontainers-based providers.B. Higher floor for setup/teardown timeouts. Container-backed fixtures do their work in setup/teardown, so floor those at 180s (
max(test_timeout, 180)) while keeping execution-timeout at the per-test value so genuinely hung tests are still caught quickly. Belt-and-suspenders for cold-cache or other slow bring-ups.Both are scoped so normal per-test hang-detection and local behavior are unchanged.
Tests: added coverage for the timeout floor and for the CI/provider-scoped pre-pull (selected / excluded / non-providers / outside-CI).
mypy-dev+ ruff clean.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines