8 items • Updated • 1
LLM-jp-3 VILA 14B
This repository provides a large vision language model (VLM) developed by the Research and Development Center for Large Language Models at the National Institute of Informatics, Japan.
Usage
Python version: 3.10.12
Clone the repository and install the libraries.
Run the python script. You can change the
image_pathandqueryto your own.
Model Details
| Model components | Model / Architecture | Parameters |
|---|---|---|
| Vision encoder | siglip-so400m-patch14-384 | 428M |
| Projector | 2-layer MLP | 32M |
| LLM | llm-jp-3-13b-instruct | 13B |
Datasets
The model was trained in three stages.
Step-0
We used the following data sets to tune the parameters in the projector.
| Language | Dataset | Images |
|---|---|---|
| Japanese | Japanese image text pairs | 558K |
| English | LLaVA-Pretrain | 558K |
Step-1
We used the following data sets to tune the parameters in the projector and LLM.
| Language | Dataset | Images |
|---|---|---|
| Japanese | Japanese image text pairs | 6M |
| Japanese interleaved data | 6M | |
| English | coyo (subset) | 6M |
| mmc4-core (subset) | 6M |
Step-2
We used the following data sets to tune the parameters in the projector and LLM.
| Language | Dataset | Images |
|---|---|---|
| Japanese | llava-instruct-ja | 156K |
| japanese-photos-conv | 12K | |
| ja-vg-vqa | 99K | |
| synthdog-ja (subset) | 102K | |
| English | LLaVA | 158K |
| VQAv2 | 53K | |
| GQA | 46K | |
| OCRVQA | 80K | |
| TextVQA | 22K |
Evaluations
We evaluated our model using Heron Bench, JA-VLM-Bench-In-the-Wild, and JA-VG-VQA500.
We used gpt-4o-2024-05-13 for LLM-as-a-judge.
Heron Bench
| Models | LLM-as-a-judge score (%) |
|---|---|
| Japanese InstructBLIP Alpha | 14.0 |
| Japanese Stable VLM | 24.2 |
| Llama-3-EvoVLM-JP-v2 | 39.3 |
| LLaVA-CALM2-SigLIP | 43.3 |
| llm-jp-3-vila-14b (Ours) | 57.2 |
| GPT-4o | 87.6 |
JA-VLM-Bench-In-the-Wild
| Models | ROUGE-L | LLM-as-a-judge score (/5.0) |
|---|---|---|
| Japanese InstructBLIP Alpha | 20.8 | 2.42 |
| Japanese Stable VLM | 23.3 | 2.47 |
| Llama-3-EvoVLM-JP-v2 | 41.4 | 2.92 |
| LLaVA-CALM2-SigLIP | 47.2 | 3.15 |
| llm-jp-3-vila-14b (Ours) | 52.3 | 3.69 |
| GPT-4o | 37.6 | 3.85 |
JA-VG-VQA-500
| Models | ROUGE-L | LLM-as-a-judge score (/5.0) |
|---|---|---|
| Japanese InstructBLIP Alpha | -- | -- |
| Japanese Stable VLM | -- | -- |
| Llama-3-EvoVLM-JP-v2 | 23.5 | 2.96 |
| LLaVA-CALM2-SigLIP | 17.4 | 3.21 |
| llm-jp-3-vila-14b (Ours) | 16.2 | 3.62 |
| GPT-4o | 12.1 | 3.58 |
Risks and Limitations
The model released in this repository is in the early stages of our research and development. It has not been tuned such that model's outputs are aligned with social norms, ethical standards, and the law.
License
The weights of this model are released under the Apache License, Version 2.0. In addition, a user of this model must comply with the OpenAI terms of use because the model used synthetic data generated by OpenAI GPT-4.
Additional information
Regarding the license of the synthdog-ja dataset, there is no explicit license statement in the dataset documentation. While we attempted to contact the main corresponding author of "OCR-free Document Understanding Transformer" for clarification, we received no response.
Based on the following considerations:
- The donut-base model trained on this dataset is released under the MIT license
- The Wikipedia articles used in the dataset are licensed under CC-BY-SA
We have determined that the synthdog-ja dataset is most likely governed by the CC-BY-SA license, and proceeded with training under this assumption.
- Downloads last month
- 11
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support