DeepSeek-R1-0528-quantized.w4a16 👁 Model Icon

👁 Validated Badge

Model Overview

• Model Architecture: DeepseekV3ForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Activation quantization: None
  • Weight quantization: INT4
• Release Date: 05/30/2025
• Version: 1.0
• Validated on: RHOAI 2.24, RHAIIS 3.2.1
• Model Developers: Red Hat (Neural Magic)
• ModelCar Storage URI: oci://registry.redhat.io/rhelai1/modelcar-deepseek-r1-0528-quantized-w4a16:1.5
• Validated on RHOAI 2.24: quay.io/modh/vllm:rhoai-2.24-cuda
• Validated on RHAIIS 3.2.1: http://registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.1
• Validated on vLLM: 0.10.0

Model Optimizations

This model was obtained by quantizing weights of DeepSeek-R1-0528 to INT4 data type. This optimization reduces the number of bits used to represent weights from 8 to 4, reducing GPU memory requirements (by approximately 50%). Weight quantization also reduces disk size requirements by approximately 50%.

Deployment

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_id = "RedHatAI/DeepSeek-R1-0528-quantized.w4a16"
number_gpus = 8
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=256)
tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "Give me a short introduction to large language model."
llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
outputs = llm.generate(prompt, sampling_params)
generated_text = outputs[0].outputs[0].text
print(generated_text)

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

We created this model using MoE-Quant, a library developed jointly with ISTA and tailored for the quantization of very large Mixture-of-Experts (MoE) models.

For more details, please refer to the MoE-Quant repository.

Evaluation

The model was evaluated on popular reasoning tasks (AIME 2024, MATH-500, GPQA-Diamond) via LightEval. For reasoning evaluations, we estimate pass@1 based on 10 runs with different seeds, temperature=0.6, top_p=0.95 and max_new_tokens=65536.

Accuracy

Base model

deepseek-ai/DeepSeek-R1-0528

(45)

September 2025 Collection of third-party generative AI models validated by Red Hat AI for use across the Red Hat AI Product Portfolio. • 4 items • Updated Apr 30 • 3