Dataset Card for The Cauldron

👁 image/png

Dataset description

The Cauldron is part of the Idefics2 release.

It is a massive collection of 50 vision-language datasets (training sets only) that were used for the fine-tuning of the vision-language model Idefics2.

Load the dataset

To load the dataset, install the library datasets with pip install datasets. Then,

from datasets import load_dataset
ds = load_dataset("HuggingFaceM4/the_cauldron", "ai2d")

to download and load the config ai2d for example.

Data fields

An example of a sample looks as follows:

{
 "images" = [PIL.Image]
 "texts" = [
 {
 "user": "Question: How many actions are depicted in the diagram?\nChoices:\nA. 6.\nB. 4.\nC. 8.\nD. 7.\nAnswer with the letter.",
 "assistant": "Answer: D",
 "source": "TQA"
 }
 ]
}

In images, there is a list of images, to be placed before the text. In texts, there is a conversation between a user and an assistant about the images that is represented by a list of turns.

Stats about the datasets in The Cauldron

Decontamination

The Cauldron contains only the train split of each sub-datasets. On top of that, we removed the few examples containing an image also present in the test splits of MMMU, MathVista or MMBench.

References to the original datasets

Licensing Information

Each of the publicly available sub-datasets present in the Cauldron are governed by specific licensing conditions. Therefore, when making use of them you must take into consideration each of the licenses governing each dataset. To the extent we have any rights in the prompts, these are licensed under CC-BY-4.0.

Citation Information

If you are using this dataset, please cite

@misc{laurençon2024matters,
 title={What matters when building vision-language models?}, 
 author={Hugo Laurençon and Léo Tronchon and Matthieu Cord and Victor Sanh},
 year={2024},
 eprint={2405.02246},
 archivePrefix={arXiv},
 primaryClass={cs.CV}
}