Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 👁 Model Icon

Model Overview

Model Architecture: Mistral3ForConditionalGeneration
- Input: Text / Image
- Output: Text
Model Optimizations:
- Activation quantization: INT8
- Weight quantization: INT8
Intended Use Cases: It is ideal for:
- Fast-response conversational agents.
- Low-latency function calling.
- Subject matter experts via fine-tuning.
- Local inference for hobbyists and organizations handling sensitive data.
- Programming and math reasoning.
- Long document understanding.
- Visual understanding.
Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages not officially supported by the model.
Release Date: 04/15/2025
Version: 1.0
Validated on: RHOAI 2.20, RHAIIS 3.0, RHELAI 1.5
Model Developers: Red Hat (Neural Magic)

Model Optimizations

This model was obtained by quantizing activations and weights of Mistral-Small-3.1-24B-Instruct-2503 to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. A combination of the SmoothQuant and GPTQ algorithms is applied for quantization, as implemented in the llm-compressor library.

Deployment

Initialize vLLM server:

vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 --tensor_parallel_size 1 --tokenizer_mode mistral

Send requests to the server:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
 api_key=openai_api_key,
 base_url=openai_api_base,
)

model = "RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16"


messages = [
 {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
 model=model,
 messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

Evaluation

The model was evaluated on the OpenLLM leaderboard tasks (version 1), MMLU-pro, GPQA, HumanEval and MBPP. Non-coding tasks were evaluated with lm-evaluation-harness, whereas coding tasks were evaluated with a fork of evalplus. vLLM is used as the engine in all cases.

Accuracy

Category	Benchmark	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8 (this model)	Recovery
OpenLLM v1	MMLU (5-shot)	80.67	80.40	99.7%
	ARC Challenge (25-shot)	72.78	73.46	100.9%
	GSM-8K (5-shot, strict-match)	56.68	61.18	104.3%
	Hellaswag (10-shot)	83.70	82.26	98.3%
	Winogrande (5-shot)	83.74	80.90	96.6%
	TruthfulQA (0-shot, mc2)	70.62	69.15	97.9%
	Average	75.03	74.56	99.4%
	MMLU-Pro (5-shot)	67.25	66.54	98.9%
	GPQA CoT main (5-shot)	42.63	44.64	104.7%
	GPQA CoT diamond (5-shot)	45.96	41.92	91.2%
Coding	HumanEval pass@1	84.70	84.20	99.4%
	HumanEval+ pass@1	79.50	81.00	101.9%
	MBPP pass@1	71.10	72.10	101.4%
	MBPP+ pass@1	60.60	62.10	100.7%
Vision	MMMU (0-shot)	52.11	53.11	101.9%
Vision	ChartQA (0-shot)	81.36	82.36	101.2%