Skip to content

Core DB tests hang on interpreter exit under Python 3.12 in CI #76135

Closed

Description

@potiuk

What happens

In the full-matrix CI runs, the Core test type in every Python 3.12 DB-core job runs all of its tests successfully, prints the pytest summary, and then the process never exits. Breeze waits until the total test timeout, terminates the pool, and the job fails with exit code 2:

3834 passed, 48 skipped, 968 deselected, 1 xfailed, 1 warning in 1224.07s (0:20:24)
...
Terminating the pool
All containers stopped. Marking the whole run as terminated on timeout
Some tasks were terminated on timeout. Please check the logs of the tasks (below) for more details.
The core test DB failed! Giving up

There is no failing test and no traceback. The other test types in the same job (API, Always, CLI, Other, Serialization) report OK.

Example timeline (Postgres 15 / 3.12, v3-4-test): Core started 07:45:47, printed its summary at 08:06:28, and was idle until the pool was killed at 08:42:20.

Scope

  • Python 3.12 only: the same jobs on 3.11, 3.13 and 3.14 pass in the same runs.
  • All DB backends: SQLite, Postgres 15, MySQL 8.4 and LowestDeps all fail this way.
  • Both main and v3-4-test. Regular main push runs only test the default Python, so they stay green; the nightly full matrix shows it.
  • Every failing job runs Python 3.12.16 (platform linux -- Python 3.12.16, pytest-9.1.1).
  • First seen on v3-4-test between the nightly of 2026-10-29 16:46 (green, dfa73fddf0) and the nightly of 2026-10-30 04:46 (red, 410177f3b2). The five commits in that range (UI, docs and e2e only) do not touch core code or the test setup, which points at the rebuilt CI image rather than the code.

Examples

Likely cause and next steps

Most likely something in the Core test session (a non-daemon thread, a multiprocessing child, or an atexit handler) keeps the interpreter from shutting down under the Python 3.12 version in the new image. To find out what is blocking:

  1. Dump the stacks at session end, e.g. arm faulthandler.dump_traceback_later(...) in pytest_unconfigure, so a CI run shows what blocks shutdown.
  2. Reproduce locally with breeze testing core-tests --test-type Core --backend sqlite --python 3.12 and run py-spy dump on the hung process.
  3. Compare against the previous 3.12 patch release by pinning the base image.

Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting

Activity

  1. added
    kind:bugThis is a clearly a bug
    area:CIAirflow's tests and continious integration
    on Oct 31, 2026
  2. potiuk commented on Oct 31, 2026

    @potiuk
    MemberAuthor

    Root cause, fixed by #76139:

    • Three triggerer tests left their forked runner process running after the test.
    • Freeze the GC before the triggerer's forks so memory stays flat #76016 freezes the GC before that fork and keeps it frozen, so in the pytest process the supervisor is never collected and the runner never sees EOF.
    • On Python 3.12.16, ResourceTracker.__del__ waits for the multiprocessing resource tracker at interpreter exit, and the leaked runners keep its pipe open, so pytest hung after its summary.

    Two corrections to the description above:

    Verified locally on the 3.12.16 CI image:


    Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:CIAirflow's tests and continious integrationkind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions