After expanding the vocabulary of the Qwen3-8B model，he predictions still contained think content, and the generation would truncate at the newly added tokens.

[J123ingjing]

在使用tokenizer.add_special_tokens({"additional_special_tokens": all_new_tokens})qwen3-8B扩充词汇表之后，在sft lora微调+embed_tokens 和 lm_head ，使用template：qwen_nothink 微调之后eval 发现predict 依旧存在think内容，并且think会中断在新添加的词汇处，设置skip_special_tokens：false，发现中断之后一直输出<end_of_text> 通过查看train/sft/workflow.py 发现additional_special_tokens_id 加入到eos_token_id 删除之后，重新sft 模型之后，进行inference/chat发现无法分析新添加的特殊词汇，不会中断，但是在胡乱think，对于新加入词汇think不会正常输出，而是在相应位置输出一些其他词汇，请问是哪一步出现了错误

[lxcxjxhx]

基于 LLaMA-Factory 源码分析，这个问题涉及三个层面：模板处理、特殊标记集成和训练配置。

1. Think 内容残留的根本原因

qwen_nothink 模板在 src/llamafactory/data/template.py 中通过 ReasoningTemplate 类实现：

class ReasoningTemplate(Template):
    @override
    def encode_oneturn(self, tokenizer, messages, system=None, tools=None):
        messages = deepcopy(messages)
        if not self.preserve_thinking:
            for i in range(1, len(messages) - 2, 2):
                messages[i]["content"] = self.remove_thought(messages[i]["content"])
        
        if self.enable_thinking is False:  # remove all cot
            messages[-1]["content"] = self.remove_thought(messages[-1]["content"])

关键点：

• preserve_thinking=False 时，会移除历史对话中的 thought 内容
• enable_thinking=False 时，会移除最后一条消息的 thought 内容
• 但这只影响训练数据的预处理，不影响模型本身的生成行为

问题所在：模型在预训练阶段已经学会了生成 <think>...</think> 的模式。即使训练数据中移除了 think 内容，模型仍可能在推理时生成这些标记，因为：

1. 模型的权重已经包含了生成 think 模式的倾向
2. LoRA 微调的参数量有限，可能不足以完全覆盖预训练学到的行为

解决方案：

template: qwen_nothink
enable_thinking: false  # 确保在推理时也禁用
# 在推理配置中显式设置
infer_backend: huggingface  # 或 vllm
generation_config:
  max_new_tokens: 512
  do_sample: true
  temperature: 0.95

2. 新添加词汇处截断的原因

当你使用 tokenizer.add_special_tokens({"additional_special_tokens": all_new_tokens}) 时：

1. Tokenizer 层面：新标记被添加到词汇表，获得新的 token ID
2. Embedding 层面：embed_tokens 和 lm_head 需要扩展以容纳新标记
3. 训练层面：新标记的 embedding 权重是随机初始化的，需要充分训练

截断问题通常发生在：

• 新标记的 embedding 权重没有充分训练
• 新标记被错误地识别为 EOS 标记
• 模型没有学会正确使用新标记

源码中的处理（src/llamafactory/data/template.py）：

def fix_special_tokens(self, tokenizer: "PreTrainedTokenizer") -> None:
    if tokenizer.eos_token_id is None:
        self._add_or_replace_eos_token(tokenizer, eos_token="<|endoftext|>")
    if tokenizer.pad_token_id is None:
        tokenizer.pad_token = tokenizer.eos_token

检查点：

• 确认新添加的标记没有被设置为 eos_token 或 pad_token
• 确认 additional_special_tokens 的 ID 范围正确

3. 正确的词汇表扩充流程

步骤 1：准备新标记

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

# 添加新标记
new_tokens = ["<custom_1>", "<custom_2>", "<custom_3>"]
tokenizer.add_special_tokens({"additional_special_tokens": new_tokens})

# 保存更新后的 tokenizer
tokenizer.save_pretrained("./qwen3-8b-extended")

步骤 2：扩展模型 embedding

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model.resize_token_embeddings(len(tokenizer))

# 保存扩展后的模型
model.save_pretrained("./qwen3-8b-extended")

步骤 3：配置训练参数

# dataset_info.json
{
  "my_dataset": {
    "file_name": "data.json",
    "formatting": "sharegpt",
    "columns": {
      "messages": "conversations"
    }
  }
}

# 训练配置
model_name_or_path: ./qwen3-8b-extended
template: qwen_nothink
finetuning_type: lora
lora_target: all
lora_rank: 16
lora_alpha: 32

# 关键：确保新标记的 embedding 被训练
train_embed: true  # 训练 embedding 层
train_lm_head: true  # 训练语言模型头

步骤 4：验证新标记的使用

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("./qwen3-8b-extended")
model = AutoModelForCausalLM.from_pretrained("./qwen3-8b-extended-lora")

# 测试新标记
input_text = "<custom_1> 这是一个测试 <custom_2>"
inputs = tokenizer(input_text, return_tensors="pt")

# 检查 token IDs
print("Token IDs:", inputs.input_ids)
print("Decoded:", tokenizer.decode(inputs.input_ids[0]))

# 生成
outputs = model.generate(**inputs, max_new_tokens=50)
print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=False))

4. 常见问题排查

问题 A：新标记无法被正确解析

原因：embedding 权重没有充分训练
解决：

• 增加训练步数
• 提高 learning rate（对于 embedding 层）
• 确保 train_embed: true

问题 B：生成在新标记处截断

原因：新标记被误识别为 EOS
解决：

# 检查 eos_token_id
print("EOS token ID:", tokenizer.eos_token_id)
print("New token IDs:", tokenizer.convert_tokens_to_ids(new_tokens))

# 确保新标记不在 eos_token_id 中
assert tokenizer.eos_token_id not in tokenizer.convert_tokens_to_ids(new_tokens)

问题 C：模型在新标记位置输出其他内容

原因：模型没有学会正确使用新标记的语义
解决：

• 在训练数据中明确展示新标记的用法
• 增加包含新标记的训练样本
• 使用全量微调（full fine-tuning）而非 LoRA

5. 推荐的完整工作流

# 1. 扩展词汇表
python scripts/expand_vocab.py \
  --model_name_or_path Qwen/Qwen3-8B \
  --new_tokens_file new_tokens.txt \
  --output_dir ./qwen3-8b-extended

# 2. 准备训练数据（包含新标记的示例）
# data.json
[
  {
    "conversations": [
      {"from": "human", "value": "<custom_1> 请分析这段内容"},
      {"from": "gpt", "value": "<custom_2> 分析结果如下..."}
    ]
  }
]

# 3. 训练（确保 embedding 层被训练）
llamafactory-cli train \
  --model_name_or_path ./qwen3-8b-extended \
  --template qwen_nothink \
  --finetuning_type lora \
  --lora_target all \
  --train_embed true \
  --train_lm_head true \
  --dataset my_dataset \
  --output_dir ./qwen3-8b-finetuned

# 4. 推理测试
llamafactory-cli chat \
  --model_name_or_path ./qwen3-8b-finetuned \
  --adapter_name_or_path ./qwen3-8b-finetuned \
  --template qwen_nothink \
  --enable_thinking false

通过以上步骤，应该能够解决 think 内容残留和新标记截断的问题。关键是确保：

1. 正确配置模板和推理参数
2. 充分训练新标记的 embedding
3. 避免新标记被误识别为特殊控制标记

[lxcxjxhx]

Based on the LLaMA-Factory source code analysis, the issue stems from two separate problems: (1) the thinking template still generating <think> content, and (2) generation stopping at newly added tokens.

Root Cause Analysis

Problem 1: Think Content Still Generated

In src/llamafactory/data/template.py, the ReasoningTemplate class controls thinking behavior through two parameters:

class ReasoningTemplate(Template):
    @override
    def encode_oneturn(self, tokenizer, messages, system=None, tools=None):
        messages = deepcopy(messages)
        if not self.preserve_thinking:
            for i in range(1, len(messages) - 2, 2):
                messages[i]["content"] = self.remove_thought(messages[i]["content"])

        if self.enable_thinking is False:  # remove all cot
            messages[-1]["content"] = self.remove_thought(messages[-1]["content"])

Key insight: Even if you use qwen_nothink template, if enable_thinking is not explicitly set to False in the template configuration, the model will still generate <think> tokens. The template name alone doesn't disable thinking—you need to ensure the template's enable_thinking attribute is False.

Problem 2: Generation Stops at New Tokens

When you use tokenizer.add_special_tokens() to add new tokens, the model's embedding layer and LM head are not automatically updated. The new tokens have random/untrained embeddings, causing the model to immediately output <end_of_text> when encountering them.

In fix_special_tokens():

def fix_special_tokens(self, tokenizer: "PreTrainedTokenizer") -> None:
    if tokenizer.eos_token_id is None:
        self._add_or_replace_eos_token(tokenizer, eos_token="<|endoftext|>")
    if tokenizer.pad_token_id is None:
        tokenizer.pad_token = tokenizer.eos_token

This method only adds EOS/PAD tokens but doesn't handle custom special tokens you add via add_special_tokens().

Solutions

Solution 1: Properly Disable Thinking in Template

Ensure your template configuration explicitly disables thinking:

# In your dataset_info.yaml or training config
template: qwen  # Use base qwen template, not qwen_nothink
# Then in the template definition, set:
enable_thinking: false
preserve_thinking: false

Or create a custom template in src/llamafactory/data/template.py:

@register_template("qwen_no_think_custom")
class QwenNoThinkCustom(ReasoningTemplate):
    def __init__(self):
        super().__init__(
            format="qwen",
            enable_thinking=False,  # Explicitly disable
            preserve_thinking=False,
        )

Solution 2: Correctly Add Special Tokens for Vocabulary Expansion

When expanding vocabulary, you must resize the model's embeddings:

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")

# Add new special tokens
new_tokens = ["<new_token_1>", "<new_token_2>"]
tokenizer.add_special_tokens({"additional_special_tokens": new_tokens})

# CRITICAL: Resize model embeddings
model.resize_token_embeddings(len(tokenizer))

# Save the updated tokenizer and model
tokenizer.save_pretrained("./expanded_vocab")
model.save_pretrained("./expanded_vocab")

Then use the expanded model for LoRA fine-tuning:

model_name_or_path: ./expanded_vocab  # Use the model with resized embeddings
template: qwen
finetuning_type: lora
lora_target: all

Solution 3: Use LLaMA-Factory's Built-in Vocabulary Expansion

LLaMA-Factory supports vocabulary expansion through the new_special_tokens parameter in model_args.py:

# training config
model_name_or_path: Qwen/Qwen3-8B
new_special_tokens: "<new_token_1>,<new_token_2>"  # Comma-separated
template: qwen
finetuning_type: lora

This automatically handles embedding resizing and initializes new token embeddings from similar existing tokens.

Solution 4: Post-Processing to Remove Think Content

If you can't retrain immediately, remove <think> content during inference:

def remove_thinking(text: str) -> str:
    import re
    # Remove all <think>...</think> blocks
    text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL)
    # Remove trailing unclosed <think>
    text = re.sub(r"<think>.*$", "", text, flags=re.DOTALL)
    return text.strip()

Verification Checklist

After applying the fixes, verify:

1. Template check: Ensure enable_thinking=False in your template
2. Embedding check: Confirm model.get_input_embeddings().num_embeddings == len(tokenizer)
3. Generation check: Test with a simple prompt to see if new tokens are generated correctly
4. LoRA check: If using LoRA, ensure lora_target includes embedding layers if you want to train them

Summary

Issue	Root Cause	Fix
Think content still generated	enable_thinking not explicitly False	Set enable_thinking: false in template
Generation stops at new tokens	Embeddings not resized after add_special_tokens()	Use model.resize_token_embeddings() or new_special_tokens parameter
<end_of_text> output	New tokens have random embeddings	Initialize embeddings properly or train them

For your use case (Qwen3-8B + vocabulary expansion + LoRA), I recommend:

1. Use new_special_tokens parameter in LLaMA-Factory config
2. Use template: qwen with enable_thinking: false
3. Verify embedding size matches tokenizer vocabulary after loading

---

If this helped resolve your issue, feel free to mark it as accepted!