show the local model's context window before the first token count by mahiatlinux · Pull Request #8882 · unslothai/unsloth

mahiatlinux · 2026-08-14T23:23:03Z

Problem

The chat header rendered the context bar only when contextUsage was set, so a resident GGUF showed nothing until a token count landed. The recount that produces that count is best-effort and stands down for images, audio, Deep Research, a pending RAG turn and a busy backend, and a chat nobody has sent has no usage at all. In all of those the context length was invisible exactly while the user was choosing a model to fit a prompt into.

Change

The bar now renders as soon as the resident model's window is known, showing — / 32.8k until a count arrives.

An uncounted chat stays distinct from a counted zero: percent remains null, so no fill, no percentage and no near-limit warning are drawn. Reporting an unmeasured prompt as 0% of the window would be the same misreport in a different place.

lib/context-window-known.ts — hasKnownContextWindow gates on the resident checkpoint rather than the picked one, so a load in flight, an evicted model and an API selection all keep their windows off the header.
lib/context-usage-bar-state.ts — deriveContextUsageBar moves the prop-to-text derivation out of the component so the node suite can drive it directly.
chat-page.tsx — the header render gate.

Screenshots

Chat header with a GGUF resident and no token count available.

Before — no bar at all:

After — the window is named, with no usage claimed:

Once a count lands, unchanged from before:

Verification

Manually verified in a browser against the real frontend, with a real llama-server serving a real GGUF (stories260K.gguf, n_ctx 2048) and token counts computed live through the same /apply-template → /tokenize pair the backend uses. Every state below was checked by hand.

tree	`/chat/count_tokens`	header
before	declines (503)	no bar
before	200	`19 / 2.0k`, fill
after	declines (503)	`— / 2.0k`, no fill
after	200	`19 / 2.0k`, fill
after	no model resident	no bar

studio/frontend/tests/context-window-visible.test.ts covers the predicate and every text state. The suite was mutation-checked: falling back to 0 instead of null, dropping the external-model guard, and dropping the percent clamp each fail it.

Frontend suite 2716 passing, tsc -b and tsc -p tsconfig.test.json clean, vite build clean.

Empty-chat count

Reported by @oobabooga on this PR (comment): a fresh chat with unsloth/Phi-4-mini-instruct-GGUF Q4_K_M resident read 1 / 131,072 at 0.0%, not -.

The recount counted an empty message list. llama.cpp renders that as the bare generation marker (<|assistant|> for this model, one token), so /chat/count_tokens returned 1 and it was published as usage.

/chat/count_tokens now refuses with 503 when the resolved prompt is empty, checked after the system prompt, the canvas instruction, the tool selection and the action nudge have been folded in. The backend is the only side that can decide it: unsloth run --enable-tools outranks a request's own enable_tools: false, so a pill-less request can still carry schemas the browser cannot see. An empty chat carrying a system prompt, project instructions, the canvas instruction or tool schemas is still counted.

Verified in a browser against the reported model with n_ctx pinned to 32768: - / 32.8k with no fill on a fresh chat, 15 / 32.8k with a system prompt set, 13 / 32.8k after one turn. Full readings in the reply.

The chat header rendered the context bar only when `contextUsage` was set, so a resident GGUF showed nothing until a token count landed. The recount that produces one is best-effort: it stands down for images, audio, Deep Research, a pending RAG turn and a busy backend, and a chat that has never been sent has no usage at all. In every one of those cases the context length was invisible exactly while the user was picking a model to fit a prompt into (unslothai#8867). The bar now renders as soon as the resident model's window is known, showing `— / 32.8k` until a count arrives. An uncounted chat is kept distinct from a counted zero: `percent` stays null, so no fill, no percentage and no near-limit warning are drawn. Reporting an unmeasured prompt as 0% of the window would be the same misreport in a new place. `hasKnownContextWindow` gates on the resident checkpoint rather than the picked one, so a load in flight, an evicted model and an API selection all keep their windows off the header. `deriveContextUsageBar` moves the prop-to-text derivation out of the component so the node suite can drive it directly.

mahiatlinux · 2026-08-15T02:53:17Z

@codex review

chatgpt-codex-connector · 2026-08-15T02:56:27Z

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 23b3dd6a6c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mahiatlinux · 2026-08-17T11:29:22Z

Before/after UI evidence for this change, from two isolated Studio installs: BEFORE at the merge base 203007d19, AFTER at head 23b3dd6a6. Both homes built from their own checked-out tree by install.sh --local, launched on separate ports, each login verified against that install's own credential.

Chat header with unsloth/gemma-4-E2B-it-GGUF UD-Q4_K_XL resident (n_ctx 131072) and nothing counted:

The same pair at full window, which is the control: same model, same variant, empty thread, and the Deep research pill armed in the composer on both halves.

How the uncounted state was reached, and why it is reproducible rather than staged: refreshContextUsage returns early on store.deepResearchEnabled before it builds a request, so arming Deep Research is a deterministic way into the state your description lists. It is armed by writing unsloth_chat_deep_research_enabled="true" to localStorage before the SPA boots, the same write setDeepResearchEnabled performs, and researchDisabled is false for a local GGUF so the composer's tools menu reaches it by hand. The armed pill is visible in the second shot on both halves.

Readings taken from the same servers that were photographed:

	BEFORE	AFTER
`resident_checkpoint`	`unsloth/gemma-4-E2B-it-GGUF`	`unsloth/gemma-4-E2B-it-GGUF`
`gguf_variant`	`UD-Q4_K_XL`	`UD-Q4_K_XL`
`context_length`	131072	131072
Deep research pill armed	yes	yes
`/chat/count_tokens` requests made by the browser	0	0
context bar present	no	yes
context bar text	—	`— / 131.1k`
context bar `aria-label`	—	`Context window: 131.1k tokens, usage not counted yet`
fill track drawn	no	no

The zero count on both sides is the part that makes the pair mean anything: the AFTER bar is drawn from the resident model's window alone, not from a count that happened to land on one side. bar_fill_present false on both sides is the second half of the change holding, no fill and no percentage for a prompt nobody measured.

Two things in the images that are not this PR: the empty-chat greeting is randomised per mount, so the halves read differently, and free GPU memory moved 9433 MiB to 7139 MiB between the two runs because another session on this box took the card. Neither touches the claim, the model was loaded with gpu_layers=0 and ran entirely on the CPU.

Downloadable copy

Same two images as a CI artifact, for anyone who wants the originals rather than the rendered inline copies: pr8882-evidence — built by run https://github.com/mahiatlinux/unsloth/actions/runs/32025370819 (green), 62,824 bytes, 30-day retention.

Artifact name: pr8882-evidence. Contents:

pr8882-header-before-after.png — 1244x104, 8,931 bytes. The chat header clip, BEFORE | AFTER.
pr8882-full-before-after.png — 1824x816, 59,634 bytes. The full window pair, the control shot.
pr8882-evidence-manifest.txt — 776 bytes. Both SHAs, the merge base and head the pair was shot at, and the run that produced the bundle.

The bundle is byte-identical to the images above: ae6bb9dd… for the header pair and b150c5c5… for the full pair, checked against the files the run photographed.

No GIF here. Nothing in this pair is a motion effect, and the run recorded static screenshots rather than a frame series, so there is no cadence to assemble honestly.

oobabooga · 2026-08-17T22:25:08Z

I can reproduce an issue with unsloth/Phi-4-mini-instruct-GGUF, using the Q4_K_M variant.

Reproduction:

Load unsloth/Phi-4-mini-instruct-GGUF (Q4_K_M).
Open a fresh chat or click New Chat.
Do not enter or send any message.
Open the Context usage display.

Actual result:

Context usage: 0.0%
Prompt tokens: 1
Completion: 0
Total: 1 / 131,072

Expected result:

Since the conversation has not started, the usage numerator should be unknown (-) rather than reporting one prompt token. The reported token appears to be BOS or chat-template overhead from counting an empty prompt, not prompt usage from the conversation.

mahiatlinux · 2026-08-18T06:24:09Z

Confirmed and fixed. Final shape is 267986f; the earlier commit put the check on the client and Codex was right that it could not see the effective tool policy, so the whole decision moved into the backend.

Cause

The recount counted whatever branch it found, including an empty one, and llama.cpp renders an empty message list as the bare generation marker. Against unsloth/Phi-4-mini-instruct-GGUF Q4_K_M on this box:

POST /apply-template  {"messages":[]}                 -> "<|assistant|>"
POST /tokenize        {"content":"<|assistant|>"}     -> [200019]
POST /api/inference/chat/count_tokens {"messages":[]} -> {"input_tokens":1}
POST /api/inference/chat/count_tokens {"messages":[{"role":"user","content":"hello"}]}
                                                      -> {"input_tokens":4}

That 1 was published as usage, so the header read 1 / 131,072 and the tooltip reported 0.0%, which is the state you hit. It predates this PR; the PR made it visible earlier by drawing the bar before the first count.

Change

/chat/count_tokens now refuses with 503 when the resolved prompt is empty. The check sits after the system prompt, the canvas instruction, the tool selection and the action nudge have all been folded in, so it fires only when nothing at all reached the template. The recount's existing catch leaves the bar in the uncounted state this PR added, — / <window>, with no fill and no percentage.

It has to be decided there rather than in the browser: unsloth run --enable-tools outranks a request's own enable_tools: false in _effective_enable_tools, so a pill-less request can still carry schemas worth thousands of tokens, and only the backend knows that. An empty chat carrying a system prompt, project instructions, the canvas instruction or tool schemas is still counted.

studio/backend/routes/inference.py holds the guard; studio/backend/tests/test_openai_auto_switch.py covers the refusal, the system-prompt chat, and the CLI-forced-tools chat.

Verification

Same model you used, unsloth/Phi-4-mini-instruct-GGUF Q4_K_M, loaded in Studio and driven in a browser. n_ctx is pinned to 32768 on both halves here, so the denominator reads 32.8k rather than your 131.1k. Nothing else about the state differs.

Fresh chat, nothing sent, same server on both halves:

	before	after
bar text	`1 / 32.8k`	`— / 32.8k`
`aria-label`	`Context usage: 1 of 32.8k tokens`	`Context window: 32.8k tokens, usage not counted yet`
tooltip	`Context usage 0.0% / Prompt tokens 1 / Completion 0 / Total 1 / 32,768`	`Context window 32,768`
fill track drawn	yes	no
`/chat/count_tokens`	200, `input_tokens` 1	503, `Cannot count tokens for an empty prompt.`

The two states that must not regress, both on the fixed build:

state	bar text	count endpoint
empty chat, system prompt set	`15 / 32.8k`, fill drawn	200, `input_tokens` 15
after one turn (`Say hi in three words.`)	`13 / 32.8k`, tooltip `prompt 9 / completion 4`	not needed, the run writes its own usage

Backend test_openai_auto_switch.py 486 passing, frontend suite 2716 passing, tsc -b and tsc -p tsconfig.test.json clean.

mahiatlinux · 2026-08-18T06:24:45Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1e528c89a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mahiatlinux · 2026-08-18T06:41:12Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 267986fc8d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mahiatlinux · 2026-08-18T06:45:46Z

@codex review

for more information, see https://pre-commit.ci

mahiatlinux · 2026-08-18T07:18:46Z

@codex review

chatgpt-codex-connector · 2026-08-18T07:22:37Z

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 511f262bf7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

wasimysaid · 2026-08-18T08:48:23Z

Went through this end to end LGTM. The context window now shows up before the first turn without pretending usage was counted. Nice fix!

Studio's chat header prices a conversation against the model's context window, and the recount behind it had a llama.cpp branch only. For an MLX-served model the endpoint answered "No GGUF model loaded", so the bar showed a dash for the whole conversation until the first reply reported its own usage -- the case unslothai#8882 made visible by rendering the bar as soon as the window is known. The endpoint now asks MLX after finding no GGUF backend, below the refusals that are about the request rather than the backend and above llama.cpp's own render concerns. A count is only worth showing if it renders what the completion for the same request would render, so this answers the questions that completion answers rather than the ones the request makes easy. Which of its two paths claims the request: the tool loop, or the relay that a client catalog or replayed tool history goes to. What each then renders: the loop's action nudge and its strip of stale call markup, the relay's rebuild that keeps structured tool_calls through templating, the launcher's tools-on default where a request stated no intent of its own and its withdrawal where the request did, the catalog a named template's tool_use branch advertises, and a zero tool budget suppressing the loop entirely. Where a helper carries one of those decisions it is called rather than restated. Three shapes are refused rather than priced short: a pending turn whose retrieval would reach the document store, since only running the search would say what it adds; an image, which /apply-template swaps for a short marker; and an empty prompt, whose bare generation marker reads as a started conversation. A vision model is counted rather than skipped. It serves text turns through the processor render its generation uses, so counting shares that render, and asks mlx_vlm which special markers tokenization should add -- a per-model answer, and on the releases from before mlx_vlm exported it, the rule those releases inline. Both store questions on this path -- the enabled MCP servers and whether retrieval can run -- open SQLite, and the second can load an extension and create the schema, so they run on the worker thread and only where their answer decides something.

mahiatlinux requested review from danielhanchen and shimmyshimmer and removed request for danielhanchen August 14, 2026 23:24

mahiatlinux self-assigned this Aug 17, 2026

stop counting an empty chat's template scaffolding as context usage

b1e528c

chatgpt-codex-connector Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread studio/frontend/src/features/chat/utils/refresh-context-usage.ts Outdated

decide the empty-prompt count where the tool policy is known

267986f

mahiatlinux requested a review from danielhanchen as a code owner August 18, 2026 06:40

chatgpt-codex-connector Bot reviewed Aug 18, 2026

View reviewed changes

Comment thread studio/backend/routes/inference.py Outdated

keep a passthrough tool catalog out of the empty-prompt refusal

b2cdecc

[pre-commit.ci] auto fixes from pre-commit.com hooks

511f262

for more information, see https://pre-commit.ci

wasimysaid merged commit 6e5360c into unslothai:main Aug 18, 2026
45 of 47 checks passed

Uh oh!

Uh oh!

Conversation

mahiatlinux commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Screenshots

Verification

Empty-chat count

Uh oh!

mahiatlinux commented Aug 15, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 15, 2026

Uh oh!

mahiatlinux commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Downloadable copy

Uh oh!

oobabooga commented Aug 17, 2026

Uh oh!

mahiatlinux commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cause

Change

Verification

Uh oh!

mahiatlinux commented Aug 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mahiatlinux commented Aug 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mahiatlinux commented Aug 18, 2026

Uh oh!

mahiatlinux commented Aug 18, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 18, 2026

Uh oh!

wasimysaid commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mahiatlinux commented Aug 14, 2026 •

edited

Loading

mahiatlinux commented Aug 17, 2026 •

edited

Loading

mahiatlinux commented Aug 18, 2026 •

edited

Loading

wasimysaid commented Aug 18, 2026 •

edited

Loading