MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
By: Jaap Jumelet , Leonie Weissweiler , Joakim Nivre and more
Potential Business Impact:
Tests computers on understanding many languages.
We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.
Similar Papers
TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs
Computation and Language
Tests computer language skills for Turkish.
UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu
Computation and Language
Tests how well computers understand Urdu grammar.
MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
Computation and Language
Tests AI's understanding of images in 205 languages.