GPT-BERT (BabyLM 10M)

Submission to the BabyLM challenge 2024 trained on Baby-cosmo-fine-10M.

The training scripts are published here: https://github.com/ltgoslo/gpt-bert

@inproceedings{charpentier-samuel-2024-bert,
 title = "{BERT} or {GPT}: why not both?",
 author = "Charpentier, Lucas Georges Gabriel and
 Samuel, David",
 editor = "Hu, Michael Y. and
 Mueller, Aaron and
 Ross, Candace and
 Williams, Adina and
 Linzen, Tal and
 Zhuang, Chengxu and
 Choshen, Leshem and
 Cotterell, Ryan and
 Warstadt, Alex and
 Wilcox, Ethan Gotlieb",
 booktitle = "The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning",
 month = nov,
 year = "2024",
 address = "Miami, FL, USA",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2024.conll-babylm.24/",
 pages = "262--283",
}

4 items • Updated May 13