Qwen3.5-4B Optimized MTPLX (Q8 trunk)

Run this with MTPLX

MTPLX is an MLX-native runtime for native Multi-Token-Prediction speculative decoding on Apple Silicon. Up to 2.24× faster decode at real coding temperatures (temp=0.6 / top_p=0.95 / top_k=20) using the model's own built-in MTP heads — no external drafter, no greedy hack.

pip install mtplx
mtplx start

Project: github.com/youssofal/MTPLX

Other MTPLX checkpoints:

Qwen3.6-27B-MTPLX-Optimized-Speed — 4-bit flagship speed (63 TPS on M5 Max)
Qwen3.6-27B-MTPLX-Optimized — verified default (GDN8-Speed4 trunk + CyanKiwi INT4 MTP)
Qwen3.5-4B-MTPLX-Optimized-Speed — small 4-bit speed-test

Q8-trunk MTPLX speed-test artifact for Apple Silicon.

This model uses the public mlx-community/Qwen3.5-4B-MLX-8bit MLX affine 8-bit trunk and grafts back the official native MTP head from Qwen/Qwen3.5-4B. The MTP head is stored as mtp.safetensors; layer-0 attention/MLP linears are quantized to 4-bit affine group-64, while mtp.fc and the MTP norms stay BF16.

Intended Use

A quick MTPLX download / load / speed-path test artifact at 4B scale, with the larger 8-bit trunk for higher-fidelity target verification. Once the runtime ships:

mtplx start

Choose Custom Hugging Face repo, then enter:

Youssofal/Qwen3.5-4B-Optimized-MTPLX

Artifact Layout

Trunk: MLX affine 8-bit, group size 64
MTP sidecar: official Qwen3.5-4B MTP tensors
MTP sidecar quantization: body-int4
Runtime contract: mtplx_runtime.json
MTPLX default: depth 2, target temperature 0.6, draft temperature 0.7

Local Smoke Result

On the local Apple Silicon MTPLX workstation, the depth-2 speed path measured 105.21 tok/s versus 75.63 tok/s AR on the warm-code prompt (max_tokens=48, temperature=0.6, top_p=0.95, top_k=20).

This Q8 artifact had the best multiplier in the local one-prompt matrix. The Q4 sibling (Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed) remains faster in absolute tok/s because its 4-bit trunk has a faster AR baseline.

Build Stats

{
 "bits": 4,
 "group_size": 64,
 "mode": "affine",
 "output_size_bytes": 86701040,
 "output_tensor_count": 29,
 "policy": "cyankiwi",
 "quantization": "body-int4",
 "quantized_linears": {
 "mtp.layers.0.mlp.down_proj": {"bits": 4, "group_size": 64, "mode": "affine"},
 "mtp.layers.0.mlp.gate_proj": {"bits": 4, "group_size": 64, "mode": "affine"},
 "mtp.layers.0.mlp.up_proj": {"bits": 4, "group_size": 64, "mode": "affine"},
 "mtp.layers.0.self_attn.k_proj":{"bits": 4, "group_size": 64, "mode": "affine"},
 "mtp.layers.0.self_attn.o_proj":{"bits": 4, "group_size": 64, "mode": "affine"},
 "mtp.layers.0.self_attn.q_proj":{"bits": 4, "group_size": 64, "mode": "affine"},
 "mtp.layers.0.self_attn.v_proj":{"bits": 4, "group_size": 64, "mode": "affine"}
 },
 "source_tensor_count": 15
}

Model tree for Youssofal/Qwen3.5-4B-Optimized-MTPLX

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(256)

this model

URL: https://huggingface.co/Youssofal/Qwen3.5-4B-Optimized-MTPLX

⇱ Youssofal/Qwen3.5-4B-Optimized-MTPLX · Hugging Face