Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset
By: Paul Lerner, François Yvon
Potential Business Impact:
Finds AI favors popular political parties when translating.
The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left, center, and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.
Similar Papers
Benchmarking Gender and Political Bias in Large Language Models
Computation and Language
Finds AI bias in political speeches and votes.
Benchmarking Gender and Political Bias in Large Language Models
Computation and Language
Finds AI bias in political speech and voting.
ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech
Computation and Language
Helps computers write speeches that sound like real politicians.