gemma-3-1b-it-quantized.w8a8

Model Overview

• Model Architecture: google/gemma-3-1b-it
  • Input: Vision-Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: INT8
  • Activation quantization: INT8
• Release Date: 6/4/2025
• Version: 1.0
• Model Developers: RedHatAI

Quantized version of google/gemma-3-1b-it.

Model Optimizations

This model was obtained by quantizing the weights of google/gemma-3-1b-it to INT8 data type, ready for inference with vLLM >= 0.8.0.

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm.assets.image import ImageAsset
from vllm import LLM, SamplingParams

# prepare model
llm = LLM(
 model="RedHatAI/gemma-3-1b-it-quantized.w8a8",
 trust_remote_code=True,
 max_model_len=4096,
 max_num_seqs=2,
)

# prepare inputs
question = "What is the content of this image?"
inputs = {
 "prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n",
 "multi_modal_data": {
 "image": ImageAsset("cherry_blossom").pil_image.convert("RGB")
 },
}

# generate response
print("========== SAMPLE GENERATION ==============")
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
print(f"PROMPT : {outputs[0].prompt}")
print(f"RESPONSE: {outputs[0].outputs[0].text}")
print("==========================================")

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created with llm-compressor by running the code snippet below:

Evaluation

The model was evaluated using lm_evaluation_harness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands:

Accuracy

Category	Metric	google/gemma-3-1b-it	RedHatAI/gemma-3-1b-it-quantized.w8a8	Recovery (%)
OpenLLM V1	ARC Challenge	36.86%	36.43%	98.84%
	GSM8K	25.17%	24.87%	98.80%
	Hellaswag	56.03%	55.62%	99.25%
	MMLU	39.99%	39.35%	98.38%
	Truthfulqa (mc2)	38.54%	38.22%	99.17%
	Winogrande	58.88%	58.96%	100.13%
	Average Score	42.58%	42.24%	99.20%

Base model

google/gemma-3-1b-pt

google/gemma-3-1b-it

(448)

Collection of quantized Gemma 3 models created by Google. • 12 items • Updated Apr 30 • 5