Qwen3.5-397B-A17B-FP8-Dynamic 👁 Model Icon

👁 Validated Badge

Model Overview

• Model Architecture: Qwen3_5MoeForCausalLM
  • Input: Text, Image
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP8
  • Activation quantization: FP8
• Release Date: 2026-03-07
• Version: 1.0
• Model Developers: RedHatAI
• ModelCar Storage URI: oci://registry.redhat.io/rhai/modelcar-qwen3-5-397b-a17b-fp8-dynamic:3.0
• Validated on vLLM: 0.18.0
• Validated on RHAIIS: 3.4
• Validated on RHOAI: 3.4

This model is a quantized version of Qwen/Qwen3.5-397B-A17B. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

Model Optimizations

This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-397B-A17B to FP8 data type, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, and token embeddings are kept in original precision.

Deployment

Use with vLLM

This model can be deployed efficiently using vLLM.

1. Text-Only: Skip the vision encoder to free up memory for additional KV cache:

vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --language-model-only --tensor-parallel-size 4

2. Multimodal (Text + Image): Serve with full vision support:

vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --tensor-parallel-size 4

3. Tool Call: Enable tool use support:

vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --tensor-parallel-size 4

4. Multi-Token Prediction (MTP): For speculative decoding:

vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' --tensor-parallel-size 4

Send requests to the server:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
 api_key=openai_api_key,
 base_url=openai_api_base,
)

model = "RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic"

messages = [
 {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
 model=model,
 messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was created by applying LLM Compressor with FP8 dynamic quantization, as presented in the code snippet below.

Evaluation

This model was evaluated on GSM8K-Platinum, MMLU-Pro, IFEval, Math 500, GPQA Diamond, AIME 25, and LiveCodeBench v6 using lm-evaluation-harness and lighteval, served with vLLM using --language-model-only.

Accuracy

Category	Benchmark	Qwen/Qwen3.5-397B-A17B	RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic	Recovery
Reasoning	GSM8K-Platinum (0-shot)	95.53	95.37	100.2%
	MMLU-Pro (0-shot)	88.27	88.33	100.1%
	Math 500 (0-shot)	84.93	84.53	99.5%
	AIME 25 (0-shot)	92.92	94.58	101.8%
	GPQA Diamond (0-shot)	89.73	88.72	98.9%
Instruction Following	IFEval prompt-level strict (0-shot)	90.63	90.70	100.1%
	IFEval inst-level strict (0-shot)	93.21	93.37	99.8%
Coding	LiveCodeBench v6 (0-shot)	83.81	83.62	99.8%

Reproduction

The results were obtained using the following commands:

Base model

Qwen/Qwen3.5-397B-A17B

(66)

May 2026 Collection of third-party generative AI models validated by Red Hat AI for use across the Red Hat AI Product Portfolio. • 10 items • Updated May 15 • 2