Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 👁 Model Icon

Model Overview

Model Architecture: Mistral3ForConditionalGeneration
- Input: Text / Image
- Output: Text
Model Optimizations:
- Weight quantization: INT4
Intended Use Cases: It is ideal for:
- Fast-response conversational agents.
- Low-latency function calling.
- Subject matter experts via fine-tuning.
- Local inference for hobbyists and organizations handling sensitive data.
- Programming and math reasoning.
- Long document understanding.
- Visual understanding.
Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages not officially supported by the model.
Release Date: 04/15/2025
Version: 1.0
Validated on: RHOAI 2.20, RHAIIS 3.0, RHELAI 1.5
Model Developers: Red Hat (Neural Magic)

Model Optimizations

This model was obtained by quantizing the weights of Mistral-Small-3.1-24B-Instruct-2503 to INT4 data type. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.

Only the weights of the linear operators within transformers blocks are quantized. Weights are quantized using a symmetric per-group scheme, with group size 128. The GPTQ algorithm is applied for quantization, as implemented in the llm-compressor library.

Deployment

Initialize vLLM server:

vllm serve RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 --tensor_parallel_size 1 --tokenizer_mode mistral

Send requests to the server:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
 api_key=openai_api_key,
 base_url=openai_api_base,
)

model = "RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16"


messages = [
 {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
 model=model,
 messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

Evaluation

The model was evaluated on the OpenLLM leaderboard tasks (version 1), MMLU-pro, GPQA, HumanEval and MBPP. Non-coding tasks were evaluated with lm-evaluation-harness, whereas coding tasks were evaluated with a fork of evalplus. vLLM is used as the engine in all cases.

Accuracy

Category	Benchmark	Mistral-Small-3.1-24B-Instruct-2503	Mistral-Small-3.1-24B-Instruct-2503-quantized.w4a16 (this model)	Recovery
OpenLLM v1	MMLU (5-shot)	80.67	79.74	98.9%
	ARC Challenge (25-shot)	72.78	72.18	99.2%
	GSM-8K (5-shot, strict-match)	58.68	59.59	101.6%
	Hellaswag (10-shot)	83.70	83.25	99.5%
	Winogrande (5-shot)	83.74	83.43	99.6%
	TruthfulQA (0-shot, mc2)	70.62	69.56	98.5%
	Average	75.03	74.63	99.5%
	MMLU-Pro (5-shot)	67.25	66.56	99.0%
	GPQA CoT main (5-shot)	42.63	47.10	110.5%
	GPQA CoT diamond (5-shot)	45.96	44.95	97.80%
Coding	HumanEval pass@1	84.70	84.60	99.9%
	HumanEval+ pass@1	79.50	79.90	100.5%
	MBPP pass@1	71.10	70.10	98.6%
	MBPP+ pass@1	60.60	60.70	100.2%
Vision	MMMU (0-shot)	52.11	50.11	96.2%
Vision	ChartQA (0-shot)	81.36	80.92	99.5%