Repository navigation
Conversation
pytest >= 8.2 instantiates every unittest TestCase method at collection and
keeps the instance until that test's teardown. 65 unit classes built their
connector source in __init__, so collection did all of that work up front,
once per xdist worker: ~10 min and 3.55 GB peak per process. The SonarCloud
ingestion nightly removes the runner's swap, so its unit job was OOM-killed
("runner has received a shutdown signal") since the pytest 9.0.3 bump.
Move that setup into setUp (collection: 574s -> 14.5s, 3.55 GB -> 1.17 GB)
and fail collection in tests/unit/conftest.py when a TestCase overrides
__init__, so the pattern does not come back.
Code Review ✅ Approved🟡 Medium risk · Moving setup across 60 ingestion unit-test classes and adding a collection guard could alter test initialization or reject valid tests. Moves unittest source construction from TestCase.init into setUp across 60 unit-test files under tests/unit, and adds a conftest guard that fails collection on any TestCase with a non-stdlib init. This cuts collection time and peak memory so the swapless nightly runner no longer runs out of memory. No issues found. VerificationRules No rules evaluated Functional validation Not enabledAuto-approval Not enabledFor more verification details, comment OptionsDisplay: compact → Showing compact verification results. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
Describe your changes:
Fixes the
py-sonarcloud-nightlyUnit Tests job, which has been killed withThe runner has received a shutdown signalsince 2026-11-05 (run 42148382276, run 41953187796).Root cause. #36641 bumped pytest from 7.0.1 to 9.0.3. From pytest 8.2, every unittest
TestCasemethod is instantiated during collection, and the instance is kept until that test's teardown; pytest 7 created each instance just before its test ran. 65 unit-test classes (60 files, mostlytests/unit/topology/**) built a whole connector source insideTestCase.__init__. As a result, collection did all of that work up front, once per xdist worker:tests/unitCI runs
-n 4, so 4 × 3.55 GB on a 16 GB runner. The nightly'sfree-disk-spacestep usesswap-storage: true, which deletes the swap, so the runner runs out of memory about 6 s into the run, with every test so far passing. PR CI keeps its swap and survives by swapping: the 3.10 unit job 127680037506 took 10 min to collect plus 39 min to run, and finished 12 s before its 60-min timeout.Fix.
__init__intosetUp, keeping the same@patchdecorators and droppingmethodName/super().__init__. unittest already creates a fresh instance for every test method, so each test sees exactly the state it saw before.setUp(Hive, Airbyte, Airbyte Cloud) get it merged into the new one. The two context keys Airbyte's oldsetUpset again are dropped.tests/unit/conftest.pynow fails collection when aTestCasegets a non-stdlib__init__, either its own or one inherited from a helper base class, so the pattern can't come back unnoticed.Type of change:
High-level design:
This is a mechanical refactor of test setup across 60 files, with no production code touched.
__init__→setUpin every unittest class undertests/unitthat did work in its constructor.pytest_pycollect_makeitemhook, which already exists intests/unit/conftest.py, rejects any newTestCaseconstructor.swap-storage: truefrom the nightly: it only hides the problem. PR CI is already one bad day away from timing out.item._instanceafter collection: it relies on pytest internals, and collection would still run every constructor.Tests:
Use cases covered
TestCasethat defines__init__(directly or through a helper base) fails collection with an actionable message.Unit tests
-n 4 --dist loadfile: 1396 passed, 1 skipped, 0 failed. An independent reviewer re-ran them with-n 6, with the same result.pytest -o testpaths=tests/unit/ --collect-only): 15,402 tests, the same count as before the change./usr/bin/time -l: 574 s / 3.55 GB peak before, 14.5 s / 1.17 GB after.cwl_airflowisn't installed (tests/unit/source/pipeline/airflow/test_connection.py). It is not related to this change.__init__override, and an__init__inherited from a helper base.setUpclass and anIsolatedAsyncioTestCase.Backend integration tests
Ingestion integration tests
tests/unit/changed, and nosrc/code changed.Follow-up:
tests/integration/superset/test_superset.pyandtests/integration/amundsen/test_metadata.pyalso do work inTestCase.__init__. Superset starts containers there for every test method, at collection. That suite's nightly jobs pass, and fixing it changes the container lifecycle, so it is left for a separate PR.Playwright (UI) tests
Manual testing performed
Swap: 0Band the PR unit job doesn't remove swap.UI screen recording:
Not applicable: test-only change.
Checklist:
Fixes <issue-number>: <short explanation>. Not applicable: this is a test fix with no issue, labelledskip-pr-checks.Fixes #<issue-number>above. Not applicable, as above.REVIEW.md(thepr-checklistskill runs it) and fixed every 🔴 finding, or answered it on the line below.91b2976b29d. No 🔴.__init__": fixed.TestCases don't override__init__") is enforced intests/unit/conftest.py, with the reason in a comment there, not only in this description.🤖 Generated with Claude Code