Run unsloth_cli/tests in Backend CI - #7598
Merged
Merged
Conversation
unsloth_cli/tests had no CI at all. unsloth_cli/** was a paths trigger and a ruff target, so the Backend CI job already fired on CLI changes but never ran these 673 tests, which cover the studio launcher, the pre-exposure gate and the auth secret writers. Four had been failing on main unnoticed. Two were stale rather than broken code: - test_studio_default_exposes_parallel_option pinned the plain --parallel default to 1, but #7455 deliberately moved _PARALLEL_DEFAULT_PLAIN to 4 so a new chat does not queue behind the previous one. Assert against the constant so the two cannot drift again. - test_reexec_forwards_api_only expected --secure --api-only to re-exec. The pre-exposure gate now refuses that combination, because api-only serves no login page and the bootstrap deadline does not apply, so a seeded password could never be changed. Drop the case and assert the refusal instead. Two only passed when a built frontend dist happened to be present, which it is not in a fresh clone or on a runner. Both reach a public-launch path where the missing-dist gate exits first, so they never got to the backend check and the run_server call they are about. Stub _find_frontend_dist the way their siblings already do. Own step rather than folding into the tests/ discovery: pyproject's testpaths is tests/, and this suite needs no PYTHONPATH or CUDA spoof, importing neither unsloth nor torch. Its deps are already installed by the job (pydantic and uvicorn, which brings click, via studio.txt; pyyaml explicitly).
This was referenced Jul 29, 2026
danielhanchen
added a commit
that referenced
this pull request
Jul 30, 2026
Three test_start.py cases assert on plain substrings of result.output, e.g. "Invalid value for '--gpu-memory-mode'". Typer renders parameter errors through Rich, which emits ANSI escapes as soon as FORCE_COLOR is set, so the substring is split across escape sequences and the assertion fails even though the message is present. They pass locally and fail in CI purely because the runner exports FORCE_COLOR; click, typer and rich are the same versions in both. Setting NO_COLOR alone does not help, FORCE_COLOR still wins, so the autouse fixture removes FORCE_COLOR and CLICOLOR_FORCE as well. #7598 only recently started running unsloth_cli/tests in Backend CI, which is why this surfaced now. Verified: the whole suite is 748 passed with and without FORCE_COLOR=1.
danielhanchen
added a commit
that referenced
this pull request
Jul 30, 2026
* Read the backend source as utf-8 in the model-picker contract test _read_backend landed in #7385 without an encoding, so it falls back to the platform default and breaks on Windows the moment a backend file it reads gains a non-ASCII byte. tests/test_source_read_encoding.py guards exactly this and is currently failing on main, which turns Repo tests (CPU) red on every open PR whose CI runs the merge commit. The sibling _read helper directly above already passes encoding = "utf-8". * Strip colour from the unsloth_cli CLI test output Three test_start.py cases assert on plain substrings of result.output, e.g. "Invalid value for '--gpu-memory-mode'". Typer renders parameter errors through Rich, which emits ANSI escapes as soon as FORCE_COLOR is set, so the substring is split across escape sequences and the assertion fails even though the message is present. They pass locally and fail in CI purely because the runner exports FORCE_COLOR; click, typer and rich are the same versions in both. Setting NO_COLOR alone does not help, FORCE_COLOR still wins, so the autouse fixture removes FORCE_COLOR and CLICOLOR_FORCE as well. #7598 only recently started running unsloth_cli/tests in Backend CI, which is why this surfaced now. Verified: the whole suite is 748 passed with and without FORCE_COLOR=1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These 673 tests ran nowhere.
unsloth_cli/**was already apathstrigger on Backend CI and a ruff target, so the job fired on CLI changes but never executed the suite that covers the studio launcher, the pre-exposure gate and the auth secret writers. Four tests had been failing onmainunnoticed.Adds a
CLI tests (unsloth_cli)step to the existingrepo-cpu-testsjob and fixes the four so the step lands green.The four failures
Two were stale tests, not broken code:
test_studio_default_exposes_parallel_option--paralleldefault to 1, but #7455 deliberately moved_PARALLEL_DEFAULT_PLAINto 4 so a new chat does not queue behind the previous one. Now asserts against the constant so the two cannot drift again.test_reexec_forwards_api_only[--secure --api-only]Two only passed when a built frontend dist happened to be present, which it is not in a fresh clone or on a runner:
test_studio_default_in_venv_broken_backend_exits_before_stripping_bootstraptest_run_in_venv_passes_secure_and_forces_hostrun_serveris called, socapturedstayed empty.Both now stub
_find_frontend_distthe way their siblings in the same files already do, making them independent of whether the frontend was built.Placement
Its own step rather than folding into the
tests/auto-discovery above it:pyproject.tomlsetstestpaths = ["tests/"], and this suite needs neither thePYTHONPATHnor theUNSLOTH_COMPILE_DISABLEspoof that step sets, since it imports neither unsloth nor torch. It self-bootstrapssys.path.No new dependencies. The job already installs everything the suite imports:
pydanticanduvicorn(which bringsclick) viastudio/backend/requirements/studio.txt, andpyyamlexplicitly. Verified by resolving each import failure in a clean venv back to a package the job already installs.Verification
python -m pytest unsloth_cli/tests -qon this branch: 673 passed, 4 skipped. Onmain: 4 failed, 669 passed.Known follow-up, not addressed here
test_studio_run_parallel_flag.pyis order-dependent: run alone it fails 6 tests that pass in a full-directory run, onmaintoday and unchanged by this PR. The CI step runs the whole directory in one invocation, which is the passing order, and I confirmed it is stable across repeated runs. Worth fixing separately, but silently papering over it inside a CI-enablement change seemed worse than naming it.