Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic 👁 Model Icon

Model Overview

Model Architecture: Mistral3ForConditionalGeneration
- Input: Text / Image
- Output: Text
Model Optimizations:
- Activation quantization: FP8
- Weight quantization: FP8
Intended Use Cases: It is ideal for:
- Fast-response conversational agents.
- Low-latency function calling.
- Subject matter experts via fine-tuning.
- Local inference for hobbyists and organizations handling sensitive data.
- Programming and math reasoning.
- Long document understanding.
- Visual understanding.
Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages not officially supported by the model.
Release Date: 04/15/2025
Version: 1.0
Validated on: RHOAI 2.20, RHAIIS 3.0, RHELAI 1.5
Model Developers: RedHat (Neural Magic)

Model Optimizations

This model was obtained by quantizing activations and weights of Mistral-Small-3.1-24B-Instruct-2503 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization.

Deployment

Initialize vLLM server:

vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 --tensor_parallel_size 1 --tokenizer_mode mistral

Send requests to the server:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
 api_key=openai_api_key,
 base_url=openai_api_base,
)

model = "RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16"


messages = [
 {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
 model=model,
 messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

Evaluation

The model was evaluated on the OpenLLM leaderboard tasks (version 1), MMLU-pro, GPQA, HumanEval and MBPP. Non-coding tasks were evaluated with lm-evaluation-harness, whereas coding tasks were evaluated with a fork of evalplus. vLLM is used as the engine in all cases.

Accuracy

Category	Benchmark	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic (this model)	Recovery
OpenLLM v1	MMLU (5-shot)	80.67	80.71	100.1%
	ARC Challenge (25-shot)	72.78	72.87	100.1%
	GSM-8K (5-shot, strict-match)	58.68	49.96	85.1%
	Hellaswag (10-shot)	83.70	83.67	100.0%
	Winogrande (5-shot)	83.74	82.56	98.6%
	TruthfulQA (0-shot, mc2)	70.62	70.88	100.4%
	Average	75.03	73.49	97.9%
	MMLU-Pro (5-shot)	67.25	66.86	99.4%
	GPQA CoT main (5-shot)	42.63	41.07	99.4%
	GPQA CoT diamond (5-shot)	45.96	45.45	98.9%
Coding	HumanEval pass@1	84.70	84.70	100.0%
	HumanEval+ pass@1	79.50	79.30	99.8%
	MBPP pass@1	71.10	70.00	98.5%
	MBPP+ pass@1	60.60	59.50	98.2%