GENDEROUS: Machine Translation and Cross-Linguistic Evaluation of a Gender-Ambiguous Dataset
- Publication type
- U
- Publication status
- Published
- Authors
- Hackenbuchner, J., Gkovedarou, EG, & Daems, J
- Series
- Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP)
- Publisher
- Association for Computational Linguistics (Vienna, Austria)
- Conference
- Network of Interdisciplinary Translation Studies in the Netherlands and Flanders (NITS) (Tilburg, The Netherlands)
- Download
- (.pdf)
- View in Biblio
- (externe link)
Abstract
Contributing to research on gender beyond the binary, this work introduces GENDEROUS, a dataset of gender-ambiguous sentences containing gender-marked occupations and adjectives, and sentences with the ambiguous or non-binary pronoun their. We cross-linguistically evaluate how machine translation (MT) systems and large language models (LLMs) translate these sentences from English into four grammatical gender languages: Greek, German, Spanish and Dutch. We show the systems’ continued default to male-gendered translations, with exceptions (particularly for Dutch). Prompting for alternatives, however, shows potential in attaining more diverse and neutral translations across all languages. An LLM-as-a-judge approach was implemented, where benchmarking against gold standards emphasises the continued need for human annotations.