Repository navigation
Description
What happens
In the full-matrix CI runs, the Core test type in every Python 3.12 DB-core job runs all of its tests successfully, prints the pytest summary, and then the process never exits. Breeze waits until the total test timeout, terminates the pool, and the job fails with exit code 2:
3834 passed, 48 skipped, 968 deselected, 1 xfailed, 1 warning in 1224.07s (0:20:24)
...
Terminating the pool
All containers stopped. Marking the whole run as terminated on timeout
Some tasks were terminated on timeout. Please check the logs of the tasks (below) for more details.
The core test DB failed! Giving up
There is no failing test and no traceback. The other test types in the same job (API, Always, CLI, Other, Serialization) report OK.
Example timeline (Postgres 15 / 3.12, v3-4-test): Core started 07:45:47, printed its summary at 08:06:28, and was idle until the pool was killed at 08:42:20.
Scope
- Python 3.12 only: the same jobs on 3.11, 3.13 and 3.14 pass in the same runs.
- All DB backends: SQLite, Postgres 15, MySQL 8.4 and LowestDeps all fail this way.
- Both
mainandv3-4-test. Regularmainpush runs only test the default Python, so they stay green; the nightly full matrix shows it. - Every failing job runs Python 3.12.16 (
platform linux -- Python 3.12.16, pytest-9.1.1). - First seen on
v3-4-testbetween the nightly of 2026-10-29 16:46 (green,dfa73fddf0) and the nightly of 2026-10-30 04:46 (red,410177f3b2). The five commits in that range (UI, docs and e2e only) do not touch core code or the test setup, which points at the rebuilt CI image rather than the code.
Examples
mainnightly: https://github.com/apache/airflow/actions/runs/41157855924/job/124410276636 (Sqlite 3.12)v3-4-test: https://github.com/apache/airflow/actions/runs/41113132731/job/124480035970 (Postgres 15 / 3.12)v3-4-testnightly: https://github.com/apache/airflow/actions/runs/41093082487 (all four 3.12 DB-core jobs)
Likely cause and next steps
Most likely something in the Core test session (a non-daemon thread, a multiprocessing child, or an atexit handler) keeps the interpreter from shutting down under the Python 3.12 version in the new image. To find out what is blocking:
- Dump the stacks at session end, e.g. arm
faulthandler.dump_traceback_later(...)inpytest_unconfigure, so a CI run shows what blocks shutdown. - Reproduce locally with
breeze testing core-tests --test-type Core --backend sqlite --python 3.12and runpy-spy dumpon the hung process. - Compare against the previous 3.12 patch release by pinning the base image.
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
Activity
Revert "[v3-4-test] Size CI test jobs on RunsOn runners to the parall…
Stop triggerer tests leaking runner processes (#76139)
[v3-4-test] Stop triggerer tests leaking runner processes (#76139) (#…
potiuk commentedon Oct 31, 2026
Root cause, fixed by #76139:
ResourceTracker.__del__waits for the multiprocessing resource tracker at interpreter exit, and the leaked runners keep its pipe open, so pytest hung after its summary.Two corrections to the description above:
v3-4-testwindow did contain the trigger. Freeze the GC before the triggerer's forks so memory stays flat #76016 reached it as58c570d519(via [v3-4-test] Freeze the GC before the triggerer's forks so memory stays flat (#76016) #76029) at 15:57 UTC on 2026-10-30.Verified locally on the 3.12.16 CI image:
executors/+test_triggerer_job.pyhang onmain, and exit with Freeze the GC before the triggerer's forks so memory stays flat #76016 reverted or with Stop triggerer tests leaking runner processes #76139.CoreDB-test run exits with Stop triggerer tests leaking runner processes #76139:3834 passed,breeze exit: 0.Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting