Skip to content

Conversation

@mudler
Copy link
Owner

@mudler mudler commented Jan 20, 2026

Description

This PR fixes models that have as part of the template the thinking token start, especially the imported models which do not ship a specific template from LocalAI. This makes it hard to discover it while processing the LLM streaming, and this PR tries to workaround that by detecting if the model has thinking tags in the template and in the first model replies.

What it does:

  • Reasoning is still extracted by expecting the full output to contain start and ending thinking tag, for instance <thinking> ... </thinking>
  • When in the template or in the messages we detect thinking start tags, we decide if we need to inject it before extracting reasoning (if it's present already we don't, if it's not present and it should, we inject it)
  • When guessing options from GGUF files, it also reads the jinja template and loads it in the model
  • It is backward compatible, in worse case models with other backends that do not provide jinja templates, won't split the reasoning from the response
  • Users can disable this behavior by setting reasoning.disable_reasoning_tag_prefill = false in the YAML configuration file of the model

Notes for Reviewers

This fixes models that are reported in #8036 #7944 and mades GLM work with LocalAI as expected.

Signed commits

  • Yes, I signed my commits.

@netlify
Copy link

netlify bot commented Jan 20, 2026

Deploy Preview for localai ready!

Name Link
🔨 Latest commit c8484b3
🔍 Latest deploy log https://app.netlify.com/projects/localai/deploys/696fbc53df282500081193b2
😎 Deploy Preview https://deploy-preview-8132--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@mudler mudler added the bug Something isn't working label Jan 20, 2026
@mudler mudler merged commit 34e054f into master Jan 20, 2026
38 checks passed
@mudler mudler deleted the fix/reasoning-template branch January 20, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant