Instella-VL-1B ✨

Welcome to the official repository for Instella-VL-1B, AMD's first ever Vision-Language Model (VLM). This repository provides a detailed guide for training and inference with Instella-VL-1B. Developed from AMD's Instella-1B (previously known as AMD OLMo 1B SFT LLM), this model is fully open-source, with both model weights and training code available for AMD GPUs (MI300). Its compact size aims to make it accessible to a broad spectrum of researchers, developers, and enthusiasts, enabling them to build upon, modify, and integrate it into their own projects.

[GitHub][Blog]

Main Results

We compare our model with models which only releases the model weights (with * in the below table) and also models which releases weights, data curvation and all training details.

DeepSeek-VL-1.3B *	InternVL2-1B *	InternVL2.5-1B *	TinyLLaVA-2.4B	TinyLLaVA-1.5B	llava-onevision-1b	MiniCPM-V-2	Instella-VL-1B
GQA	--	55.06	56.66	61.58	60.28	57.95	--	61.52
SQA	64.52	89.54	93.90	64.30	59.69	59.25	76.10	83.74
POPE	85.80	87.40	89.95	85.66	84.77	87.17	86.56	86.73
MM-Bench	64.34	61.70	68.40	58.16	51.28	44.60	70.44	69.17
seedbench	65.94	65.90	71.30	63.30	60.04	65.43	66.90	68.47
MMMU	28.67	32.40	35.60	32.11	29.89	30.90	38.55	29.30
realworldqa	50.20	51.90	58.30	52.42	46.67	51.63	55.03	58.82
mmstar	38.30	46.18	47.93	37.17	31.87	37.38	40.93	43.21
Average	-	61.26	65.26	56.84	53.06	54.29	-	62.62
ocrbench	41.40	74.40	74.20	28.90	34.40	43.00	60.00	67.90
TextVQA	57.54	69.60	72.96	47.05	49.54	49.54	74.23	71.23
AI2D	51.13	62.40	67.58	49.58	43.10	57.35	64.40	66.65
ChartQA	47.40	71.52	75.76	12.96	15.24	61.24	59.80	72.52
DocVQA	35.70	80.94	82.76	25.82	30.38	71.22	69.54	80.30
InfoVQA	20.52	46.30	53.62	21.35	24.46	41.18	38.24	46.40
OCR Average	42.28	67.53	71.15	30.94	32.85	53.92	61.04	67.50

Quick Start

Follow below packages list for setting up the inference environment.

pip==25.0
wheel==0.45.1
setuptools==75.8.0
torch==2.6.0
torchvision==0.21.0
transformers==4.49.0
einops==0.8.0

import torch
from transformers import AutoTokenizer, AutoProcessor, AutoConfig, AutoModelForCausalLM

from PIL import Image
import requests
from io import BytesIO

def load_image(image_file):
 if image_file.startswith("http") or image_file.startswith("https"):
 response = requests.get(image_file)
 image = Image.open(BytesIO(response.content)).convert("RGB")
 else:
 image = Image.open(image_file).convert("RGB")
 return image


config = AutoConfig.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("amd/Instella-VL-1B", config=config, trust_remote_code=True)
processor = AutoProcessor.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("amd/Instella-VL-1B", trust_remote_code=True).to('cuda') # or 'cpu'
model.eval()
 
# For single image and text
query="Describe the image."
image=load_image("path/to/your_image") # can be a https:// url
out = processor.encode(query, image, model.get_vision_tower().image_processor, tokenizer, config)
inputs = {k: v.to(model.device) for k, v in out.items() if isinstance(v, torch.Tensor)}
with torch.inference_mode():
 output_ids = model.generate(inputs["input_ids"], images=inputs['image_tensor'], image_sizes=out['image_sizes'], do_sample=True, num_beams=1, temperature=0.2, max_new_tokens=1024, use_cache=True, stopping_criteria=out['stopping_criteria'], eos_token_id=out['eos_token_id'])
outputs = processor.decode(output_ids)
print("InstellaVL: ", outputs)

# For batch of images and text.
query=["Describe the image.", "What is the color of the dog?"]
image=[load_image("../assets/images/instellavl.png"), load_image("../assets/images/example2_dog.jpg")]
outs = processor.batch_encode(query, image, model.get_vision_tower().image_processor, tokenizer, config)

for idx, o in enumerate(outs):
 ins = {k: v.to(model.device) for k, v in o.items() if isinstance(v, torch.Tensor)}
 with torch.inference_mode():
 output_ids = model.generate(ins["input_ids"],
 images=ins['image_tensor'],
 image_sizes=o['image_sizes'],
 do_sample=True,
 num_beams=1,
 temperature=0.2,
 max_new_tokens=1024,
 use_cache=True,
 stopping_criteria=o['stopping_criteria'],
 eos_token_id=o['eos_token_id'])
 outputs = processor.decode(output_ids)
 print("Query: ", query[idx])
 print("InstellaVL: ", outputs)

Model Architecture

We initialize the vision encoder from CLIP-ViT-L/14@336 and initialize LM from AMD OLMo 1B SFT

Training Stages

Hardware

Training was conducted with up to 4 nodes, totaling 32 GPUs. Each node comprises 8 AMD Instinct™ MI300X GPUs

MLP warmup: 1 node
Pretraining: 2 nodes
Finetune: 4 nodes

Datasets

MLP Warmup

BLIP558K

Pretraining Stage

Instruction-tuning Stage

Further, to strengthen model’s understanding of science-based and general reasoning questions, as identified through error analysis, we oversampled (almost doubled the volume) specific datasets from the SFT dataset pool as detailed below.

Oversampled (~2x sampling rate): ScienceQA, AI2D, PMC-VQA, Cambrian, and TQA

Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.

For the details of training hyperparameters, please check our github repo

Contributors

Core contributors: Ximeng Sun, Aditya Kumar Singh, Gowtham Ramesh, Zicheng Liu

Contributors: Pratik Prabhanjan Brahma, Ze Wang, Jiang Liu, Jialian Wu, Prakamya Mishra, Xiaodong Yu, Yusheng Su, Sudhanshu Ranjan, Emad Barsoum

Bias, Risks, and Limitations

This model is made accessible without any safety guarantees. Users should be aware that the model may generate outputs that are sensitive, inaccurate, harmful, biased, or otherwise objectionable based on user prompts. It is crucial for users to conduct comprehensive safety evaluations, implement safety filtering, and verify the model's outputs to mitigate these risks.

License

See Files for license and any notices.

Citing

@misc{Instella-VL-1B, 
 title = {Instella-VL-1B: First AMD Vision Language Model}, 
 url = {https://huggingface.co/amd/Instella-VL-1B}, 
 author = {Ximeng Sun, Aditya Singh, Gowtham Ramesh, Jiang Liu, Ze Wang, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Prakamya Mishra, Jialian Wu, Xiaodong Yu, Yusheng Su, Emad Barsoum, Zicheng Liu}, 
 month = {March}, 
 year = {2025} 
} 

1 item • Updated Mar 2 • 1