REAP-pruned & quantized Qwen3.5 / 3.6 / Coder variants. • 15 items • Updated
Support this work → · X · GitHub · REAP paper · Cerebras REAP
Qwen3.5-99B
REAP-pruned Qwen/Qwen3.5-122B-A10B.
At a glance
| Base model | Qwen/Qwen3.5-122B-A10B |
| Format | BF16 |
| Total params | 99B |
| Active / token | 10B |
| Experts / layer | — |
| Layers | — |
| Hidden size | — |
| Context | — |
| On-disk size | 198 GB |
Which variant should I pick?
| Variant | Format | Link |
|---|---|---|
Qwen3.5-264B |
BF16 | link |
Qwen3.5-264B-FP8 |
FP8 | link |
Qwen3.5-264B-W4A16 |
W4A16 | link |
Qwen3.5-28B |
BF16 | link |
Qwen3.5-35B-EXL3-4bpw |
EXL3-4bpw | link |
Qwen3.5-76B |
BF16 | link |
Qwen3.5-76B-GGUF |
GGUF | link |
Qwen3.5-88B |
BF16 | link |
Qwen3.5-99B (this) |
BF16 | link |
Qwen3.5-99B-GGUF |
GGUF | link |
20% expert-pruned variant of Qwen3.5-122B-A10B using REAP (Routing-Enhanced Activation Pruning).
Model Details
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen3.5-122B-A10B |
| Architecture | Qwen3.5 MoE (GDN + Full Attention) |
| Original Experts | 256 per layer |
| Pruned Experts | 205 per layer (20% removed) |
| Active Parameters | ~10B per token |
| Pruning Method | REAP with targeted refusal preservation |
| Preserve Threshold | 80% (super-expert protection) |
| Calibration | reap-calibration-data-v1 — 23k benchmark-free samples |
| Maintainer | 0xSero |
| Organization | Sybil Solutions |
| Project | REAP PR17 |
Benchmark Results
Code Generation (EvalPlus)
| Benchmark | Pass@1 |
|---|---|
| HumanEval (base) | 81.1% |
| HumanEval+ (base + extra) | 76.8% |
| MBPP (base) | 86.2% |
| MBPP+ (base + extra) | 73.0% |
Knowledge & Reasoning (lm-eval, 0-shot)
| Task | Baseline | REAP-20 | Retained |
|---|---|---|---|
| arc_challenge | 63.4% | 63.7% | 100.5% |
| boolq | 86.4% | 82.7% | 95.8% |
| hellaswag | 85.9% | 84.1% | 97.9% |
| mathqa | 68.5% | 67.3% | 98.1% |
| mmlu_world_religions | 91.2% | 86.0% | 94.2% |
| openbookqa | 46.4% | 45.6% | 98.3% |
| piqa | 83.8% | 82.3% | 98.1% |
| truthfulqa_mc2 | 51.9% | 52.4% | 100.8% |
| winogrande | 75.6% | 75.5% | 99.9% |
Average capability retained: 97.9% after removing 20% of experts.
Usage
vllm serve 0xSero/Qwen3.5-99B \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--max-model-len 8192 \
--trust-remote-code \
--language-model-only \
--dtype bfloat16
Important: Use --language-model-only flag — this is a text-only checkpoint pruned from the multimodal base model.
What is REAP?
REAP (Routing-Enhanced Activation Pruning) removes the least-activated experts from MoE models while preserving critical capabilities. It uses router activation patterns from a calibration dataset to identify dispensable experts, with special protection for safety-critical behaviors.
License
Same license as the base model (Qwen).
License & citation
License inherited from the base model.
@misc{lasby2025reap,
title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
year = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}
Sponsors
Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.
- Downloads last month
- 64
Safetensors
Model size
99B params
Tensor type
BF16
·