Skip to content

Fix OOM-killed ingestion unit nightly: build test sources in setUp, not TestCase.__init__ - #37246

Open
ulixius9 wants to merge 2 commits into
mainfrom
ulixius9/failing-test-check
Open

ulixius9 wants to merge 2 commits into
mainfrom
ulixius9/failing-test-check

Conversation

@ulixius9

@ulixius9 ulixius9 commented Nov 6, 2026

Copy link
Copy Markdown
Member

Describe your changes:

Fixes the py-sonarcloud-nightly Unit Tests job, which has been killed with The runner has received a shutdown signal since 2026-11-05 (run 42148382276, run 41953187796).

Root cause. #36641 bumped pytest from 7.0.1 to 9.0.3. From pytest 8.2, every unittest TestCase method is instantiated during collection, and the instance is kept until that test's teardown; pytest 7 created each instance just before its test ran. 65 unit-test classes (60 files, mostly tests/unit/topology/**) built a whole connector source inside TestCase.__init__. As a result, collection did all of that work up front, once per xdist worker:

pytest 7.0.1 (Nov 4 nightly) pytest 9.0.3 (before this PR) this PR
Collection of tests/unit ~2 min ~10 min (CI), 574 s locally 14.5 s locally
Peak memory of one collecting process — 3.55 GB 1.17 GB

CI runs -n 4, so 4 × 3.55 GB on a 16 GB runner. The nightly's free-disk-space step uses swap-storage: true, which deletes the swap, so the runner runs out of memory about 6 s into the run, with every test so far passing. PR CI keeps its swap and survives by swapping: the 3.10 unit job 127680037506 took 10 min to collect plus 39 min to run, and finished 12 s before its 60-min timeout.

Fix.

  • Move the work from __init__ into setUp, keeping the same @patch decorators and dropping methodName / super().__init__. unittest already creates a fresh instance for every test method, so each test sees exactly the state it saw before.
  • The classes that already had a setUp (Hive, Airbyte, Airbyte Cloud) get it merged into the new one. The two context keys Airbyte's old setUp set again are dropped.
  • tests/unit/conftest.py now fails collection when a TestCase gets a non-stdlib __init__, either its own or one inherited from a helper base class, so the pattern can't come back unnoticed.

Type of change:

  • Bug fix

High-level design:

This is a mechanical refactor of test setup across 60 files, with no production code touched.

  • Approach: __init__ → setUp in every unittest class under tests/unit that did work in its constructor.
  • Guard: a pytest_pycollect_makeitem hook, which already exists in tests/unit/conftest.py, rejects any new TestCase constructor.
  • Alternatives rejected:

Tests:

Use cases covered

  • The ingestion unit suite collects in seconds with bounded memory under pytest 9, so the swapless nightly runner no longer runs out of memory.
  • A new TestCase that defines __init__ (directly or through a helper base) fails collection with an actionable message.

Unit tests

  • Converted files: all 60 pass locally with pytest 9.0.3 and -n 4 --dist loadfile: 1396 passed, 1 skipped, 0 failed. An independent reviewer re-ran them with -n 6, with the same result.
  • Full collection (pytest -o testpaths=tests/unit/ --collect-only): 15,402 tests, the same count as before the change.
    • Measured with /usr/bin/time -l: 574 s / 3.55 GB peak before, 14.5 s / 1.17 GB after.
    • The one collection error is local only: cwl_airflow isn't installed (tests/unit/source/pipeline/airflow/test_connection.py). It is not related to this change.
  • Guard: probed with throwaway test files.
    • Fails: a direct __init__ override, and an __init__ inherited from a helper base.
    • Passes: a plain setUp class and an IsolatedAsyncioTestCase.
  • Not run locally: the full unit-suite execution. PR CI runs it.

Backend integration tests

  • Not applicable (no backend changes).

Ingestion integration tests

  • Not applicable. Only tests/unit/ changed, and no src/ code changed.

Follow-up: tests/integration/superset/test_superset.py and tests/integration/amundsen/test_metadata.py also do work in TestCase.__init__. Superset starts containers there for every test method, at collection. That suite's nightly jobs pass, and fixing it changes the container lifecycle, so it is left for a separate PR.

Playwright (UI) tests

  • Not applicable (no UI changes).

Manual testing performed

  1. Pulled the logs of the failing nightly runs:
    • Nov 5 and 6 (pytest 9.0.3) die about 6 s after collection ends, with the runner shutdown signal.
    • Nov 4 (pytest 7.0.1) collected in 2 min and passed.
  2. Confirmed the nightly unit job runs with Swap: 0B and the PR unit job doesn't remove swap.
  3. Measured collection time and peak memory locally before and after the fix (table above).

UI screen recording:

Not applicable: test-only change.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>. Not applicable: this is a test fix with no issue, labelled skip-pr-checks.
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above. Not applicable, as above.
  • I have commented on my code, particularly in hard-to-understand areas. The guard explains why it exists.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed. Not applicable.
  • For a UI feature or feature task: Docker recording. Not applicable.
  • I have added tests (unit / integration / Playwright as applicable) and listed them above.
  • Self-review: I reviewed the final diff against REVIEW.md (the pr-checklist skill runs it) and fixed every 🔴 finding, or answered it on the line below.
    • Reviewed SHA 91b2976b29d. No 🔴.
    • 🟡 "guard misses an inherited __init__": fixed.
    • 🟡 "integration tests use the same pattern": deferred, see the follow-up above.
  • Decision check: no record is needed. The rule ("unit TestCases don't override __init__") is enforced in tests/unit/conftest.py, with the reason in a comment there, not only in this description.

🤖 Generated with Claude Code

pytest >= 8.2 instantiates every unittest TestCase method at collection and
keeps the instance until that test's teardown. 65 unit classes built their
connector source in __init__, so collection did all of that work up front,
once per xdist worker: ~10 min and 3.55 GB peak per process. The SonarCloud
ingestion nightly removes the runner's swap, so its unit job was OOM-killed
("runner has received a shutdown signal") since the pytest 9.0.3 bump.

Move that setup into setUp (collection: 574s -> 14.5s, 3.55 GB -> 1.17 GB)
and fail collection in tests/unit/conftest.py when a TestCase overrides
__init__, so the pattern does not come back.
@ulixius9
ulixius9 requested a review from a team as a code owner November 6, 2026 08:48
@ulixius9
ulixius9 requested a review from stout-yeoman November 6, 2026 08:48
@ulixius9 ulixius9 added the skip-pr-checks Bypass PR metadata validation check label Nov 6, 2026
Copilot AI balanced review requested due to automatic review settings November 6, 2026 08:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Nov 6, 2026
@gitar-bot

gitar-bot Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Code Review ✅ Approved

🟡 Medium risk · Moving setup across 60 ingestion unit-test classes and adding a collection guard could alter test initialization or reject valid tests.

Moves unittest source construction from TestCase.init into setUp across 60 unit-test files under tests/unit, and adds a conftest guard that fails collection on any TestCase with a non-stdlib init. This cuts collection time and peak memory so the swapless nightly runner no longer runs out of memory. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

For more verification details, comment gitar display:verbose.

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs skip-pr-checks Bypass PR metadata validation check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants