Model Description

Qwen2.5-VL-7B-Instruct-Finetuned-Os-Atlas is a GUI grounding model finetuned from Qwen/Qwen2.5-VL-7B-Instruct.

Evaluation Results

We evaluated our model using Screenspot on two benchmarks: Screenspot Pro and Screenspot v2.

We also include evaluation scripts used on these benchmarks. The table below compares our model's performance against the base model performance.

Note - The base model scores slightly lower than the mentioned scores in the paper because the prompts used for evaluation are not publicly available. We used the default prompts when evaluating the base and fine-tuned models.

Training procedure

This model used the OS-Copilot dataset for fine-tuning: OS-Copilot.

👁 Visualize in Weights & Biases

This model was trained with SFT and LoRA.

Evaluation Scripts:

Evaluation scripts available here - Screenspot_Qwen2.5_VL

Quick Start

from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
# default: Load the model on the available device(s)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
 "Fintor/Qwen2.5-VL-7B-Instruct-Finetuned-Os-Atlas", 
 torch_dtype=torch.bfloat16,
 attn_implementation="flash_attention_2",
 device_map="auto",
)
# default processer
processor = AutoProcessor.from_pretrained("Fintor/Qwen2.5-VL-7B-Instruct-Finetuned-Os-Atlas")
# Example input
messages = [
 {
 "role": "user",
 "content": [
 {
 "type": "image",
 "image": "path/to/image.jpeg",
 },
 {"type": "text", "text": "Describe this image."},
 ],
 }
]
# Preparation for inference
text = processor.apply_chat_template(
 messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
 text=[text],
 images=image_inputs,
 videos=video_inputs,
 padding=True,
 return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
 out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
 generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

Citation

Base model

Qwen/Qwen2.5-VL-7B-Instruct

(1104)