Qwen2.5-VL-3B — Medical VQA (VQA-RAD)

This is a fine-tuned version of Qwen/Qwen2.5-VL-3B-Instruct trained on the VQA-RAD radiology visual question answering dataset using QLoRA. The LoRA adapter weights have been merged into the base model, so no PEFT dependency is required at inference time.

Model Details

---

Training Details

LoRA Target Modules

Language attention and MLP layers. Vision encoder weights were frozen during training — only the language decoder was adapted.

Dataset

VQA-RAD (flaviagiammarino/vqa-rad) is a clinician-generated radiology VQA dataset with 2,248 question-answer pairs across 315 radiology images covering chest X-rays, head CTs, and abdominal scans. Questions are split into two types: - Closed-ended: Yes/No and constrained categorical answers - Open-ended: Free-form short phrase answers Evaluation is on the official test split (451 samples).

Evaluation Results

Overall

By Question Type (Fine-tuned Model)

Note on metrics: Closed-ended EM and F1 are identical because yes/no answers are single tokens. The open-ended EM→F1 gap (24.50% → 34.75%) indicates that many EM failures are phrasing mismatches rather than factual errors — the model produces semantically correct answers that don't exactly match the ground truth string.

---

Usage

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image
import torch
 
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
 "vishal98m/qwen2.5-vl-3b-finetuned-medical-vqa-rad",
 torch_dtype=torch.bfloat16,
 device_map="auto"
)
processor = AutoProcessor.from_pretrained(
 "Qwen/Qwen2.5-VL-3B-Instruct",
 use_fast=False
)
 
def ask(image: Image.Image, question: str) -> str:
 prompt = (
 f"{question}\n"
 "Provide only the short direct answer in 1-5 words. "
 "Do not explain or add any extra text."
 )
 messages = [
 {
 "role": "user",
 "content": [
 {"type": "image", "image": image},
 {"type": "text", "text": prompt},
 ],
 }
 ]
 text = processor.apply_chat_template(
 messages, tokenize=False, add_generation_prompt=True
 )
 image_inputs, video_inputs = process_vision_info(messages)
 inputs = processor(
 text=[text],
 images=image_inputs,
 videos=video_inputs,
 padding=True,
 return_tensors="pt",
 ).to("cuda")
 
 with torch.no_grad():
 generated_ids = model.generate(**inputs, max_new_tokens=32)
 
 trimmed = [
 out[len(inp):]
 for inp, out in zip(inputs.input_ids, generated_ids)
 ]
 return processor.batch_decode(
 trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
 )[0].strip()
 
# Example
image = Image.open("chest_xray.jpg")
print(ask(image, "Is there any abnormality in this chest X-ray?"))

---

Inference Tips

• Prompt format matters: The model was trained to produce short answers. Prompting with "Provide only the short direct answer in 1-5 words" significantly improves exact match performance by suppressing verbose outputs.
• Normalization: For evaluation, lowercase and strip punctuation before comparing answers.
• Hardware: Runs on a single 8GB+ VRAM GPU in bfloat16. For CPU inference, switch to torch_dtype=torch.float32.

---

Intended Use

✅ Suitable For

• Medical radiology question answering (academic and educational contexts)
• Study aid for medical and life science students
• Generating explanations of radiological findings and imaging concepts
• Prototype development for medical education tools

❌ Not Suitable For

• Clinical decision-making or patient diagnosis
• Replacing licensed medical professionals
• Providing personalised medical advice
• Any safety-critical medical application

---

Limitations

• Domain: Fine-tuned exclusively on VQA-RAD. Generalization to other medical VQA benchmarks (PathVQA, SLAKE) has not been evaluated.
• Open-ended answers: Token F1 (34.75%) is a more faithful measure of open-ended performance than EM (24.50%) — strict string matching penalizes valid paraphrases.
• Not for clinical use: This model is a research artifact. It should not be used for clinical decision-making or patient care.
• Model size: At 3B parameters, absolute performance is below larger medical VLMs. Larger variants would likely yield higher scores.
• Vision encoder frozen: Only the language decoder was fine-tuned. Fine-tuning the vision encoder may yield additional gains.

---

Citation

If you use this model, please cite the original VQA-RAD dataset:

@article{lau2018dataset,
 title={A Dataset of Clinically Generated Visual Questions and Answers about Radiology Images},
 author={Lau, Jason J and Gayen, Soumya and Ben Abacha, Asma and Demner-Fushman, Dina},
 journal={Scientific data},
 volume={5},
 number={1},
 pages={1--10},
 year={2018},
 publisher={Nature Publishing Group}
}

And the relevant methods:

@article{hu2022lora,
 title={LoRA: Low-Rank Adaptation of Large Language Models},
 author={Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
 journal={ICLR},
 year={2022}
}
 
@article{dettmers2023qlora,
 title={QLoRA: Efficient Finetuning of Quantized LLMs},
 author={Dettmers, Tim and Pagnoni, Artidoro and Fansi, Ari and Zettlemoyer, Luke},
 journal={NeurIPS},
 year={2023}
}

Uploaded model

• Developed by: vishal98m
• License: apache-2.0
• Finetuned from model : unsloth/Qwen2.5-VL-3B-Instruct

This qwen2_5_vl model was trained 2x faster with Unsloth

👁 Image

Base model

Qwen/Qwen2.5-VL-3B-Instruct

(216)