Qwen3.5-4B-MTP-GGUF

Qwen3.5-4B from Alibaba's Qwen team is a compact 4B-parameter dense multimodal reasoning model with a native 262K context window (extensible to 1M+ tokens), featuring a hybrid Gated DeltaNet + Gated Attention architecture, 248K vocabulary for 201 languages, multi-token prediction, and early-fusion training for unified text, image, and video understanding. Despite its small size, it achieves remarkable intelligence—27 on the Artificial Analysis Intelligence Index, outperforming all models under 5B parameters including Falcon-H1R-7B and NM-Nano-9B-V2—while delivering 65.4% on MMMU-Pro multimodal reasoning, 77 on the Intelligence Index, and comparable performance to GPT-OSS-20B on complex reasoning at roughly 5× fewer parameters. Apache 2.0-licensed and optimized for consumer hardware (~3GB VRAM in 4-bit quantization), it excels at agentic workflows with native tool calling, OCR, document parsing, visual question answering, and code generation, supporting vLLM/Ollama/llama.cpp for edge-to-server deployment as the most capable sub-5B multimodal model available.

Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.

Model Files

Quants Usage

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):

👁 image.png

Base model