studio: opt-in model auto-switch for the image and video apis by mahiatlinux · Pull Request #8766 · unslothai/unsloth

mahiatlinux · 2026-08-14T02:16:43Z

Closes part of #8752.

POST /v1/images/generations answered 503 unless a model had already been picked on the Images page, and its model field was documented as informational, so an API client could not choose what it generated on. Chat has had this since openai_api_auto_switch_model; media had no equivalent, and media_auto_unload_idle_seconds could unload a pipeline but never load one.

What it does

core/inference/media_auto_switch.py resolves the requested name against the downloaded image and video models, waits for in-flight work to drain, then runs the same load /images/load and /video/load run.

Names come from the local-model scan the Images picker already uses, so a repo id, the scanner id, a display label, and <id>:<QUANT> for a GGUF all resolve. Two cache-layout details matter: a cached GGUF loads by repo id rather than its snapshot directory, whose entries are symlinks into blobs/ that the loader's containment check rejects, and a cached pipeline loads from snapshots/<sha>, which is where its model_index.json lives.

POST /api/inference/video/generate gains an optional model for the same purpose. The Video page never sends it, so that route is unchanged for existing callers.

Behaviour

Gated on media_api_auto_switch_model, off by default, with its own toggle under Settings > API. With it off nothing here runs and model keeps its informational meaning, so existing clients see no change.

Nothing starts a download. A name that resolves to no downloaded model is refused with the ids that would work, rather than answered by whichever model happens to be resident, so a caller never gets one model's output under another's name.

Both waits are bounded because secure mode's tunnel caps an origin response near 100 seconds. A drain that does not clear in 30s returns 409; a load still running after 90s returns 503 with Retry-After, leaving the load to finish for the retry. A load that fails returns 503 carrying the loader's own reason instead of a bare 500.

One coupled change

"Only unload models loaded by the API" used to zero the media TTL outright, on the grounds that nothing but the user could load an image or video model. Auto-switch breaks that premise, so it is now a per-model rule: the load routes record who asked, and media_keepwarm spares a user-loaded pipeline while still collecting an API-loaded one, matching what chat already does with _loaded_by_user_action. With auto-switch off every resident model is user-loaded, so the outcome is unchanged.

Testing

Verified end to end against a running backend on Apple Silicon (MPS) with a real Z-Image Turbo GGUF: cold auto-load then generate, swap to the full pipeline and back, a quant-qualified name, a repeat request that does not reswap, an unknown name refused with the downloaded ids, and the video route refusing with a plain detail rather than an OpenAI envelope. /images/load and /api/inference/images/generate were re-checked for regressions, and with the setting off a named model is ignored exactly as before.

Automated: 21 new backend tests in tests/test_media_auto_switch.py covering resolution (diffusers directory, GGUF repo bare and per quant, standalone .gguf, both HF cache layouts, task keying, absolute paths) and the switch (inert while off, unknown name refused, already-serving, companion base_repo not counted as serving, drain timeout, load timeout, load failure, both route wirings). tests/test_media_keepwarm.py gains the per-model provenance rule, and tests/media-auto-switch-setting.test.ts covers the settings round-trip. Backend ruff, frontend tsc, i18n parity and the 2351-test frontend suite all pass.

Not covered

The OpenAI-shaped /v1 video route from #8752 is not part of this. Audio already has /v1/audio/speech and /v1/audio/transcriptions, and image has /v1/images/generations; video is still only on /api/inference/video/*.

POST /v1/images/generations answered 503 unless a model had already been picked on the Images page, and its `model` field was documented as informational, so an agent could not choose what it generated on. Chat has had this since openai_api_auto_switch_model; media had no equivalent, and media_auto_unload_idle_seconds could unload a pipeline but never load one. core/inference/media_auto_switch.py resolves the requested name against the downloaded image and video models, waits for in-flight work to drain, then runs the same load /images/load and /video/load run. Names come from the local-model scan the Images picker already uses, so a repo id, the scanner id, a display label, and `<id>:<QUANT>` for a GGUF all resolve. A cached GGUF loads by repo id rather than its snapshot directory, whose entries are symlinks into blobs/ that the loader's containment check rejects; a cached pipeline loads from snapshots/<sha>, which is where its model_index.json lives. POST /api/inference/video/generate gains an optional `model` for the same purpose. The Video page never sends it, so that route is unchanged for existing callers. Gated on media_api_auto_switch_model, off by default, with its own toggle under Settings > API. With it off nothing here runs and `model` keeps its informational meaning. Nothing starts a download, so a name that resolves to no downloaded model is refused with the ids that would work rather than answered by whichever model happens to be resident. Both waits are bounded because secure mode's tunnel caps an origin response near 100 seconds: a drain that does not clear in 30s returns 409, and a load still running after 90s returns 503 with Retry-After, leaving the load to finish for the retry. "Only unload models loaded by the API" changes with it. That setting used to zero the media TTL outright, on the grounds that nothing but the user could load an image or video model. Auto-switch breaks that premise, so it is now a per-model rule: the load routes record who asked, and media_keepwarm spares a user-loaded pipeline while still collecting an API-loaded one, matching what chat already does with _loaded_by_user_action.

for more information, see https://pre-commit.ci

Refuse a pick whose companions are not on disk. The resolver only indexes downloaded checkpoints, but a GGUF loads its text encoders and VAE from a base repo the loader prefetches, so a request could pull tens of gigabytes despite the setting promising it never downloads. The same planner /images/download-plan serves now answers first, and a nonzero remainder is a 409 naming the missing gigabytes. A plan that cannot be built is treated as nothing missing: that is almost always an unreachable Hub, where no download can happen either. Discount requests queued on the switch lock from the drain. Two concurrent requests for the same absent model are both counted by the media middleware, so the lock holder saw the queued one as work to drain, waited the full 30 seconds and returned 409, and so did the other. The chat switch already excludes its own waiters for this reason. Give the drain and the load one shared budget. Separate 30s and 90s allowances added up to roughly 120s, past the ~100s origin window the bounds exist to stay inside, so a slow switch lost the socket instead of returning the retryable 503. The drain is now capped inside a 90s end-to-end budget and the load inherits what is left. Match a GGUF on quant in the serving check. Loose .gguf files in one scan folder share that folder as their model_path, so the path alone reported a sibling as already serving and the API generated with the wrong weights. The quant is the per-file identity the backend publishes, and the same one media_keepwarm counts as part of the build. Key load provenance to the target it was recorded for. A load is recorded when it is accepted and can still fail during prefetch with the previous model resident, so an API load that failed marked a user-loaded pipeline as API-loaded and "only unload models loaded by the API" then evicted it. An origin now only answers for the model it was written against. Hold the keep-warm gate from the final drain check through load registration. Nothing stopped a generation entering that gap, and the load path cancels active work as it tears the pipeline down, so a swap that had just waited for the queue to clear could cut one short.

Plan the download against the engine that will load the pick, not the resident one. With native sd.cpp active and a diffusers pipeline requested, the sd.cpp planner refuses the pick and that refusal read as nothing missing, so the guard let the load download after all. predict_engine picks the planner, as /images/download-plan already does. Hold the admission gate across the final drain observation instead of after it. A request admitted between a passing drain and the gate is tracked but has not marked the backend active, so it read as idle and the load's teardown could still cancel it. The drain condition is now evaluated once more under the gate. Recheck the target after draining. A retry can acquire the switch lock while the earlier attempt's load is still running; draining waits that out, and without this the retry tore down the model that had just landed and reloaded it. Confirm the requested model actually served. _await_loaded returned success as soon as anything was resident, so a user load accepted between two polls superseded ours and the request generated on the replacement while naming the requested model. The final status is matched against the pick, GGUF quant included.

Compare a GGUF on the variant lister's full label rather than the backend's quant token. The token collapses IQ4_XS-3.53bpw and IQ4_XS-3.97bpw to IQ4_XS and reduces an unlabelled file to nothing, so either build reported the other as already serving. A label that cannot match the published token now reloads instead of risking the wrong weights. Key load provenance by the build, not the path. A user-loaded Q4 and an API load of Q8 from one repo shared a target string, so a failed API load marked the resident Q4 as API-loaded and "only unload models loaded by the API" freed it. Refuse when locality cannot be established. video.download_plan returns zero bytes on any metadata error because its own caller falls back to an inline pull, so the guard read that as a complete cache and allowed the download it exists to prevent. The planner now flags the failure and the switch treats an unverifiable plan as a refusal. Re-resolve the image engine under the admission gate. A concurrent load can activate the other engine while this request drains, leaving the captured backend idle and both checks passing against something nothing is using. Start the budget before resolution and bound the scan and the plan with it. It began after the cold scan and never covered the planner's Hub calls, so either could consume the response window before a deadline was consulted. Serve an exact resident match without consulting the index. A scan that failed caches an empty result, which turned a request naming the loaded model into a 404 for that window.

The previous round compared a GGUF against the variant lister's full label, which no backend publishes: status derives gguf_variant with extract_quant_token, so IQ4_XS-3.97bpw and every unlabelled checkpoint never matched and the switch returned 503 for a model that had loaded correctly. The comparison is back on the published token, and the case that motivated the label is handled where it belongs: a pick whose token another build under the same path also publishes is marked ambiguous at index time and never short-circuits, so the switch reloads instead of assuming. The load a request started is checked without that rule, since it is its own load and nothing is being assumed. Drop an alias two models answer to. Cached repos advertise their final component, so org-a/model and org-b/model both offer "model", and binding whichever the scan reached first would load arbitrary weights for a name the resolver documents as usable. Full ids are unaffected. Keep the exact-resident shortcut away from a GGUF. A bare repo id means the preferred quant, which a repo id comparison cannot see, so a resident non-preferred quant answered the request. Normalize a bare single-file directory before planning. Both load routes reinterpret it as a single_file load and resolve that family's companions, while the plan described a local pipeline with nothing to fetch. Treat plan entries as missing, not just their byte total. Both planners keep an entry whose size could not be read and coerce that size to zero, so a pending multi-GB fetch reported as zero bytes passed the guard. Flag a partial image plan. _estimate_download_bytes swallows a companion metadata failure and returns whatever it had accumulated, so a plan missing its companions looked complete; it now records the failure and download_plan reports plan_failed, which the switch already refuses. Plan on the card the load will rank for itself, since automatic precision can select a different hosted pre-quantized artifact per card. Re-plan under the admission gate. The drain can last 30 seconds, and a cache deletion during it sees a target that is neither loaded nor loading, so files verified before the wait could be gone by the time the load starts.

Flagging a partial image plan in the previous round turned every local full pipeline into a refusal: the planner asks HfApi about the absolute path, that failure is now recorded, and the switch reads it as unverifiable. A directory on disk is what from_pretrained loads, so it is complete by definition and is no longer planned at all. Verify the diffusers fallback when the native engine is predicted. predict_engine treats an absent sd.cpp binary as available whenever its installation is allowed, while activation falls back to diffusers if that install produces nothing runnable, and the two engines read different companion sets. Both plans are checked when they can differ. Compare filesystem paths without folding case. /models/Foo and /models/foo are different models where the filesystem says so, and folding them reported one as already serving the other. Aliases and repo ids still fold. Bound the post-drain re-plan by the switch budget. It runs while the admission gate is held, so a stalled Hub call held every new generation and load off that backend as well as overrunning the response window.

Verify the diffusers fallback only when the prediction assumed an install. select_and_activate_engine falls back to diffusers when no runnable sd.cpp binary exists, so on a host that already has one the load stays native and demanding the diffusers shards refused a model sd.cpp can serve, including after an idle unload. predict_engine's own binary probe is extracted as native_binary_installed so both callers share one implementation. Skip partial catalog rows. A cancelled or incomplete pull still lists, and registering it advertised an id whose load fails predictably and reported it in the "not found" listing. Detect the HF cache layout rather than the scanner's source label. A cache tree inside a user-added scan folder is relabelled custom, while its snapshot entries stay symlinks into blobs/ that the loader's containment check refuses, so those rows have to load by repo id like any other cached repo. Refuse an edit-only family before switching. The local catalog tags Kontext and Qwen-Image-Edit text-to-image, so the switch loaded a multi-GB pipeline that this endpoint then rejected for lacking txt2img, with the previously useful model already evicted.

Locale parity runs strict in CI, so the three new keys are translated into every overlay rather than left to the English fallback. mediaIdlePaused is restored with its new wording, since the setting it named no longer vetoes the media TTL outright. Refuse an incompatible plan before switching. A FLUX.2 GGUF paired with a different-size base is fully cached and still unloadable, and the route's cheap validation misses it, so the mismatch surfaced only from the background loader once the resident pipeline was already torn down. Bound load setup by the switch budget. Companion preflight and a first-run sd.cpp install both run before begin_load registers, while the admission gate is held, so a stall blocked every new generation and load on that backend as well as overrunning the response window. Preserve case in the resident shortcut. It compared a filesystem path case-insensitively, which the _satisfied_by comparison had already stopped doing. Do not advertise a directory the load route rejects. Several checkpoints and no model_index.json is ambiguous, and both routes refuse rather than choose, so indexing it only cost a failed switch.

Shield load setup from the deadline. Cancelling the await does not stop the worker, so a timeout landing inside select_and_activate_engine left the engine switched, the resident model unloaded and no begin_load coming, with the gate already released. The setup now always reaches registration; the timeout only frees the caller. Acquire the switch lock within the budget. A request that spent most of it resolving could queue behind another full switch and pass the response window before any inner wait noticed. Flag the native MiniMax-H3 plan failure. That planner has its own metadata-error return, which the switch read as a verified cache and would have downloaded the missing components. Compare the H3 partition. A switch sends no h3_task and therefore loads the default keyframe denoiser, so a resident ref2va does not answer a plain request: serving it accepted a generation that then failed for missing references. Accept modular pipeline indexes in discovery. A fully downloaded dense MiniMax-H3 carries modular_model_index.json, and rejecting it 404'd every named request for it. Rank root variants for a bare id. A repo holding both root and subdirectory builds could hand a bare id a qualified build, where a plain local load resolves non-recursively and the picker and chat resolver both take the root. Keep case in provenance keys, matching the other filesystem comparisons.

Shielding kept the setup running but not the contexts around it: a timeout unwound the admission gate and the switch lock while the task was still short of begin_load, so a newly admitted generation could start on an engine that setup was about to activate or tear down. The gated section is now its own task that acquires the gate and releases the lock itself, so giving up on the wait frees only the caller and nothing else is admitted until registration completes. Check the MiniMax-H3 partition in the resident shortcut too. It returns before _resident_is_pick runs, so a resident ref2va still answered a plain request for that repo id and the generation then failed for missing references.

Cancel a shielded future rather than closing it. An already-expired budget reached the shield branch and raised AttributeError, since a Future has no close(), instead of the retryable 503 the branch exists to return. Bound the final plan inside the gated task. It holds the admission gate and the switch lock, so a stalled planner blocked every generation and load on that backend indefinitely while the caller had long since returned. The step has no side effects, so timing out can give both back. Count a request as a waiter only while it waits for the switch lock. It stayed marked through load polling, so a second switch discounted a request that was about to generate and could replace its model out from under it. Ignore requests parked behind the gate the task owns. A newcomer is counted pending and then blocks on that same gate, so counting it aborted an otherwise idle switch with model_busy. Normalize paths in the ambiguity scan and record the MiniMax-H3 partition in provenance, so a case-sensitive filesystem stops merging distinct directories and a failed API load of the default partition no longer unpins a resident ref2va the user loaded.

…ckpoint note_load_origin never grew the partition parameter both load routes had started passing, so every /video/load raised TypeError after its background load was already accepted. It takes and stores the partition now, and a route-level test drives the real load route so a signature that drifts from its callers fails here rather than in production. Derive the expected MiniMax-H3 partition from the selected checkpoint. The native backend publishes ref2va for a minimax_h3_ref2va denoiser, so assuming the keyframe default rejected the checkpoint that had just loaded and reloaded a resident one that already matched. Discount every recorded switch waiter. The holder leaves the marker when it acquires the lock, so subtracting one fewer left a phantom request and the holder drained the full window before returning busy against an idle backend. Plan local video pipelines rather than trusting the directory. A local MiniMax-H3 modular pipeline still substitutes a hosted quantized conditioner during assembly, tens of gigabytes the shortcut would have waved through. Record a GGUF variant only for a gguf load. A single-file checkpoint with a precision token in its name stored a variant that status never publishes for that kind, so the idle policy read the mismatch as unknown and never freed an API-loaded pipeline.

… token The native sd.cpp status reports a GGUF through dtype and a quant rather than model_kind, so the resident shortcut treated every native checkpoint as a plain pipeline and served whichever quant happened to be up instead of the preferred one a bare id names. Send the resolved MiniMax-H3 partition with the load. The route records what the request carried while the backend publishes what it resolved, so leaving it unset gave every API-loaded H3 pipeline an origin key that never matched and the idle policy never freed it. Thread the caller's HF token into planning and loading. A cached gated repo resolves by repo id, so the planner still makes authenticated metadata reads; without the token they failed and the switch refused a model whose files were all local. Only the caller's header is used, never the server owner's ambient credential.

for more information, see https://pre-commit.ci

from_single_file(config = <repo id>, subfolder = "transformer") is a Hub resolution, not a local read: diffusers 0.39 pops local_files_only and forwards it into the load_config() that resolves that id, and an unset flag is None, which permits the network. The pipeline assembly after it was already guarded, so this was the last unguarded Hub read on the GGUF/safetensors image and video paths, and it runs after the resident pipeline has been evicted. (cache_dir does not help: diffusers forwards it to the checkpoint read only, never to that load_config.) The flag alone is not enough. transformer/config.json was excluded from the staged base file set on both paths, correctly for the shards, which the checkpoint supplies, but that leaves the config systematically absent rather than absent in a race: the locality gate clears the pick and local_files_only would then fail essentially every API-initiated GGUF load. So the config, and only the config, is now admitted to the base file set. A complete install loads offline as promised; an install missing it is refused with a 409 before eviction instead of losing the resident model to a late LocalEntryNotFound. The two in-tree tests that pinned the blanket exclusion are updated to assert the intended shape (config in, shards out).

for more information, see https://pre-commit.ci

chatgpt-codex-connector · 2026-08-16T01:44:54Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 583c8333fe

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

danielhanchen · 2026-08-16T05:01:39Z

Consolidated record of the review fixes on this branch, since the round-by-round threads have been pruned.

Eight defects were found and fixed, each with a regression test that fails on the unfixed source. Five are the same shape: the no-download contract holds only if every loader on the path honours local_files_only, and one missed link breaks it.

Area	Defect
`sd_cpp_backend.py`	`begin_load` had no `local_files_only` parameter, so every native (sd.cpp) image load raised `TypeError`. CPU-only hosts, opted-in MPS, and `UNSLOTH_DIFFUSION_ENGINE=sd_cpp`, including the UI path, with the new setting off.
`diffusion.py`	`_load_dense_quant_pipeline` referenced `local_files_only` without it being a parameter: `NameError` on GGUF plus `transformer_quant`.
`video.py`	The whole `_run_load` staging phase (estimate, prequant probes, checkpoint fetch, TE fetches, base predownload) ran unguarded, so an API video load could pull multi-GB after eviction.
`diffusion.py`	Same leak on the image path: `_prefetch_files`, the byte estimate, the pre-cast plan and the base preflight.
`media_locality.py`	Dangling symlinks counted as downloaded, so locality could clear a model that was not really present.
`media_auto_switch.py`	Gate ordering: the chat lifecycle gate is now taken before the media admission gates, matching chat's own switch, so a parked media request is not read as running chat work.
Family assemblers	Krea 2, LTX 2.3, HiDream and the H3 conditioner build every component themselves and never see the guarded `pipe_kwargs`; each reached the Hub on an API load. Also pinned to the live HF cache root, since the locality gate reads the live root and an unpinned lookup resolves through huggingface_hub's import-time constant.
`diffusion.py`, `video.py`	`from_single_file` received `sf_kwargs` without the flag while `config` is a repo id, which diffusers resolves through `load_config`. Fixed together with admitting `transformer/config.json` to the staged base file set: the flag alone would have turned a silent 1 KB fetch into a hard failure on essentially every API-initiated GGUF load.

Two Codex items were declined rather than fixed, with reasoning: the teardown-window generation race (already fenced by _teardown_waiters; the proposed remedies would hang or 409 unrelated users), and the keep-warm provenance overwrite (the mismatch branch deliberately errs toward sparing a resident model, and the exemption lasts until the next load rather than permanently).

Verification: full backend suite on both host shapes against a like-for-like baseline, no new failures and none accidentally fixed; ubuntu-latest 1512 passed; macOS and the Mac Studio and Windows GGUF legs green.

The red checks on this PR are not from this branch. Repo tests (CPU) and the Python 3.10-3.13 jobs fail the same way on main's own runs, including on a docs-only commit, and the remaining entries are runs cancelled by a superseding push.

mahiatlinux requested a review from danielhanchen as a code owner August 14, 2026 02:16

[pre-commit.ci] auto fixes from pre-commit.com hooks

fd5c884

for more information, see https://pre-commit.ci

This comment was marked as resolved.

Sign in to view

unslothai deleted a comment from chatgpt-codex-connector Bot Aug 14, 2026

This comment was marked as resolved.

Sign in to view

unslothai deleted a comment from mahiatlinux Aug 15, 2026

unslothai deleted a comment from chatgpt-codex-connector Bot Aug 15, 2026

unslothai deleted a comment from mahiatlinux Aug 15, 2026

unslothai deleted a comment from chatgpt-codex-connector Bot Aug 15, 2026

[pre-commit.ci] auto fixes from pre-commit.com hooks

647a4f2

for more information, see https://pre-commit.ci

This comment was marked as resolved.

Sign in to view

danielhanchen and others added 2 commits August 15, 2026 22:31

[pre-commit.ci] auto fixes from pre-commit.com hooks

583c833

for more information, see https://pre-commit.ci

Uh oh!

Uh oh!

Conversation

mahiatlinux commented Aug 14, 2026

What it does

Behaviour

One coupled change

Testing

Not covered

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

chatgpt-codex-connector Bot commented Aug 16, 2026

Uh oh!

danielhanchen commented Aug 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants