Community recipe: domain-specific SFT+DPO on Qwen2.5 → GGUF → Ollama (Yggdrasil v1-v9)

[rudi193-cmd]

What I built

Fine-tuned a domain-specific tool-calling model (Yggdrasil) through 9 iterations using Unsloth. Sharing the recipe in case it's useful to others doing similar domain-knowledge injection.

Base models tried: Qwen2.5-3B-Instruct (v1), then Functionary-small-v3.1 (v2-v9)
Platform: Kaggle (free GPU tier)
Training notebook: https://github.com/rudi193-cmd/yggdrasil-training-data

Key finding: DPO-only ≠ knowledge injection

The biggest lesson from v1-v4: DPO-only training shapes behavioral preferences but cannot inject factual domain knowledge. After DPO-only training, the model still identified itself as "Alibaba Cloud" and had zero knowledge of the domain it was being trained for.

The fix: SFT first to inject facts, then DPO on top to shape behavior. This is probably obvious to ML practitioners but wasn't obvious to me from the docs.

Recipe that worked (v7+)

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments

# 1. Load with Unsloth (4x faster, 2x less VRAM)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meetkai/functionary-small-v3.1",
    max_seq_length=2048,
    load_in_4bit=True,
)

# 2. Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=True,
)

# 3. SFT on domain facts FIRST
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=sft_dataset,  # ~2k domain Q&A pairs
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        num_train_epochs=3,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=10,
        output_dir="outputs",
        optim="adamw_8bit",
    ),
)
trainer.train()

# 4. Export to GGUF (Q4_K_M)
model.save_pretrained_gguf("yggdrasil", tokenizer, quantization_method="q4_k_m")

Deployment

# Push to HuggingFace Hub
# Then pull directly into Ollama:
ollama pull hf.co/rudi193/yggdrasil-v9:Q4_K_M
ollama cp hf.co/rudi193/yggdrasil-v9:Q4_K_M yggdrasil:v9

Results

• v1-v4 (DPO-only): model had zero domain knowledge, was very fast
• v7-v9 (SFT+DPO): model correctly answered domain questions, good tool-calling behavior
• Q4_K_M GGUF: ~1.9GB, runs comfortably on 16GB RAM

Thanks to Unsloth for making the training loop fast enough to iterate 9 times in a reasonable timeframe. The 4x speedup on Kaggle's T4 was the difference between this being viable or not.