Papers
arxiv:2504.02768
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
Published on Apr 3, 2025
Abstract
MultiBLiMP 1.0 evaluates large language models across 101 languages and 6 linguistic phenomena using a large dataset of minimal pairs, revealing challenges in modeling low-resource languages.
Generated by Qwen/Qwen2.5-Coder-32B-Instruct
We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages, 6 linguistic phenomena and containing more than 125,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.
Get this paper in your agent:
hf papers read 2504.02768