You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Kodeseer-9B

A LoRA fine-tuned Qwen3.5-9B model for GUI element grounding — predicting the (x, y) coordinates of UI elements from screenshots given natural language instructions.

Results

ScreenSpot-V2 Breakdown

ScreenSpot-Pro Full Breakdown (1581 samples)

Comparison with State-of-the-Art

ScreenSpot-V2

ScreenSpot-Pro

*MVP is a training-free inference trick

ScreenSpot Original

Usage

import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from peft import PeftModel
from PIL import Image

base_model = "Qwen/Qwen3.5-9B"
adapter = "mdabis/qwen35-9b-gui-grounding-v1"

model = Qwen3_5ForConditionalGeneration.from_pretrained(
 base_model, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter, subfolder="checkpoint-3100")
model.eval()

image = Image.open("screenshot.png").convert("RGB")
instruction = "Click the submit button"

messages = [
 {"role": "system", "content": (
 "You are a GUI grounding assistant. Given a screenshot and a user instruction, "
 "return the exact coordinates of the target UI element using the format: "
 "<|box_start|>(x,y)<|box_end|> where x and y are in [0, 1000] range."
 )},
 {"role": "user", "content": [
 {"type": "image", "image": image},
 {"type": "text", "text": instruction},
 ]},
]

text = processor.apply_chat_template(
 messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

with torch.no_grad():
 output_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)

generated = output_ids[:, inputs["input_ids"].shape[1]:]
response = processor.batch_decode(generated, skip_special_tokens=False)[0]
print(response)
# Example output: <|box_start|>(512,340)<|box_end|>

Coordinate Format

The model predicts click coordinates in <|box_start|>(x,y)<|box_end|> format where x and y are in [0, 1000] range. To convert to pixel coordinates:

pixel_x = int(x / 1000 * image_width)
pixel_y = int(y / 1000 * image_height)

Limitations

• Trained on English instructions only
• Weakest on niche professional software (AutoCAD 41.2%, FruitLoops 40.4%, Origin 38.7%)
• SFT-only — no RL/GRPO applied yet (further gains expected)
• No training data from professional software domains (all training data is general desktop/mobile/web)

License

Apache 2.0 (same as base model)

Base model