gemma-3n-E4B-it-FP8-Dynamic 👁 Model Icon

👁 Validated Badge

Model Overview

• Model Architecture: gemma-3n-E4B-it
  • Input: Audio-Vision-Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP8
  • Activation quantization: FP8
• Release Date: 08/01/2025
• Version: 1.0
• Validated on: RHOAI 2.24, RHAIIS 3.2.1
• Model Developers: RedHatAI
• ModelCar Storage URI: oci://registry.redhat.io/rhelai1/modelcar-gemma-3n-e4b-it-fp8-dynamic:1.5
• Validated on RHOAI 2.24: quay.io/modh/vllm:rhoai-2.24-cuda
• Validated on RHAIIS 3.2.1: http://registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.1
• Validated on vLLM: 0.10.0

Quantized version of google/gemma-3n-E4B-it.

Model Optimizations

This model was obtained by quantizing the weights of google/gemma-3n-E4B-it to FP8 data type, ready for inference with vLLM >= 0.10.0

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm.assets.image import ImageAsset
from vllm import LLM, SamplingParams

# prepare model
llm = LLM(
 model="RedHatAI/gemma-3n-E4B-it-FP8-Dynamic",
 trust_remote_code=True,
 max_model_len=4096,
 max_num_seqs=2,
)

# prepare inputs
question = "What is the content of this image?"
inputs = {
 "prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n",
 "multi_modal_data": {
 "image": ImageAsset("cherry_blossom").pil_image.convert("RGB")
 },
}

# generate response
print("========== SAMPLE GENERATION ==============")
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
print(f"PROMPT : {outputs[0].prompt}")
print(f"RESPONSE: {outputs[0].outputs[0].text}")
print("==========================================")

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created with llm-compressor by running the code snippet below.

Evaluation

The model was evaluated using lm_evaluation_harness for OpenLLM V1 and V2 text-based benchmarks. The evaluations were conducted using the following commands:

Accuracy

Category	Metric	google/gemma-3n-E4B-it	FP8 Dynamic	Recovery (%)
OpenLLM V1	arc_challenge	60.24	59.04	98.01%
	gsm8k	60.12	70.81	117.79%
	hellaswag	74.94	73.28	97.79%
	mmlu	64.14	64.82	101.06%
	truthfulqa_mc2	54.87	54.61	99.53%
	winogrande	68.35	67.72	99.08%
	Average	63.78	65.05	101.99%
Leaderboard	bbh	55.46	55.20	99.53%
	mmlu_pro	34.38	34.28	99.71%
	musr	33.20	34.26	103.19%
	ifeval	84.41	83.93	99.43%
	gpqa	30.87	31.38	101.65%
	math_hard	45.54	46.60	102.33%
	Average	47.31	47.61	100.63%

Base model

google/gemma-3n-E4B

google/gemma-3n-E4B-it

(60)

September 2025 Collection of third-party generative AI models validated by Red Hat AI for use across the Red Hat AI Product Portfolio. • 4 items • Updated Apr 30 • 3