Qwen3.5-2B-MTP-GGUF

Qwen3.5-2B from Alibaba's Qwen team is a compact 2B-parameter dense multimodal language model with native vision encoder, featuring a hybrid Gated DeltaNet + Gated Attention architecture (24 layers, hidden dim 2048, 8 Q heads / 2 KV heads with 256 head dim, 64 RoPE dim), 248K vocabulary for 201 languages, multi-token prediction, and a massive 262K native context window (extensible to 1M+ tokens via YaRN) for unified text, image, and video understanding. It achieves 16 on the Artificial Analysis Intelligence Index—outperforming all sub-2B models like phi-4-mini-instruct (14) and Llama-3.2-3B (15)—while delivering 78.6% IFEval, 66.5% MMLU-Pro (with thinking mode), 84.5% OCRBench, and 75.6% VideoMME at ~4.25GB VRAM (1.5GB in 4-bit), making it ideal for edge deployment. Apache 2.0-licensed with toggleable "thinking" mode for step-by-step reasoning, vLLM/Ollama/llama.cpp support, and native tool calling, it excels at agentic workflows, OCR, document parsing, visual QA, frontend coding, and multilingual chatbots as the intelligent "sweet spot" between 0.8B and 4B for local AI on consumer hardware.

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

Quants Usage

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):

👁 image.png

Base model