gemma-3-4b-it-quantized.w4a16

Model Overview

• Model Architecture: google/gemma-3-4b-it
  • Input: Vision-Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: INT4
  • Activation quantization: FP16
• Release Date: 6/4/2025
• Version: 1.0
• Model Developers: RedHatAI

Quantized version of google/gemma-3-4b-it.

Model Optimizations

This model was obtained by quantizing the weights of google/gemma-3-4b-it to INT4 data type, ready for inference with vLLM >= 0.8.0.

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from transformers import AutoProcessor

# Define model name once
model_name = "RedHatAI/gemma-3-4b-it-quantized.w4a16"

# Load image and processor
image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

# Build multimodal prompt
chat = [
 {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the content of this image?"}]},
 {"role": "assistant", "content": []}
]
prompt = processor.apply_chat_template(chat, add_generation_prompt=True)

# Initialize model
llm = LLM(model=model_name, trust_remote_code=True)

# Run inference
inputs = {"prompt": prompt, "multi_modal_data": {"image": [image]}}
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))

# Display result
print("RESPONSE:", outputs[0].outputs[0].text)

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created with llm-compressor by running the code snippet below:

Evaluation

The model was evaluated using lm_evaluation_harness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands:

Accuracy

Category	Metric	google/gemma-3-4b-it	RedHatAI/gemma-3-4b-it-quantized.w4a16	Recovery (%)
OpenLLM V1	ARC Challenge	56.57%	56.57%	100.00%
	GSM8K	76.12%	72.33%	95.02%
	Hellaswag	74.96%	73.35%	97.86%
	MMLU	58.38%	56.33%	96.49%
	Truthfulqa (mc2)	51.87%	50.81%	97.96%
	Winogrande	70.32%	68.82%	97.87%%
	Average Score	64.70%	63.04%	97.42%
Vision Evals	MMMU (val)	39.89%	40.11%	100.55%
	ChartQA	50.76%	49.32%	97.16%
	Average Score	45.33%	44.72%	98.86%

Base model

google/gemma-3-4b-pt

google/gemma-3-4b-it

(466)

Collection of quantized Gemma 3 models created by Google. • 12 items • Updated Apr 30 • 5