Skip to content

feat: add mini Qwen3-VL model for local e2e RL testing#1919

Open
samsja wants to merge 1 commit intomainfrom
feature/mini-qwen3-vl
Open

feat: add mini Qwen3-VL model for local e2e RL testing#1919
samsja wants to merge 1 commit intomainfrom
feature/mini-qwen3-vl

Conversation

@samsja
Copy link
Member

@samsja samsja commented Mar 1, 2026

Summary

  • Adds scripts/mini_vlm.py — creates a tiny ~44M param Qwen3-VL model with random weights for local e2e RL testing, following the same pattern as scripts/mini_moe.py
  • Updates is_vlm_model() to also detect VLM models from local paths by reading config.json model_type, so mini/custom checkpoints work without matching the Qwen/Qwen3-VL* naming convention

Mini model specs

Component Config
Text hidden=256, intermediate=512, layers=4, heads=4, kv_heads=2, head_dim=64
Vision hidden=128, intermediate=256, depth=4, heads=4, out_hidden=256
Total ~44M parameters

Usage

# Create and verify
uv run python scripts/mini_vlm.py --output-dir ./mini-qwen3-vl

# Verify only
uv run python scripts/mini_vlm.py --output-dir ./mini-qwen3-vl --verify-only

Test plan

  • Script creates model, saves with tokenizer+processor
  • Verification passes: text-only and multimodal forward passes produce correct logit shapes
  • is_vlm_model() detects local path models via config.json model_type
  • Existing tests unaffected

🤖 Generated with Claude Code


Note

Medium Risk
Touches VLM detection used in model loading/orchestration; misclassification or unexpected config.json contents could route runs down VLM-specific loading/freezing paths. The rest is an additive local testing script.

Overview
Adds scripts/mini_vlm.py to create a small random-weight Qwen3-VL checkpoint, copy a compatible tokenizer/processor, and sanity-check both text-only and multimodal forward passes on CPU.

Updates is_vlm_model() to recognize local/custom checkpoints by reading config.json and checking model_type (in addition to the existing Qwen/Qwen3-VL* name pattern), enabling VLM-specific codepaths to work with mini models stored on disk.

Written by Cursor Bugbot for commit 951b442. This will update automatically on new commits. Configure here.

Add scripts/mini_vlm.py that creates a tiny ~44M param Qwen3-VL model
with random weights for local testing (text: 256 hidden, 4 layers;
vision: 128 hidden, 4 depth). Follows the same pattern as mini_moe.py.

Also update is_vlm_model() to detect VLM models from local paths by
reading config.json model_type, so mini models don't need to match
the Qwen/Qwen3-VL* naming pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant