Literary machine translation under the magnifying glass : assessing the quality of an NMT-translated detective novel on document level

Publication type
P1
Publication status
Published
Authors
Fonteyne, M., Tezcan, A., & Macken, L.
Series
PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION (LREC 2020)
Pagination
3783-3791
Publisher
European Language Resources Association (ELRA)
Conference
12th International Conference on Language Resources and Evaluation (LREC) (Marseille, France)
Download
(.pdf)
View in Biblio
(externe link)

Abstract

Several studies (covering many language pairs and translation tasks) have demonstrated that translation quality has improved enormously since the emergence of neural machine translation systems. This raises the question whether such systems are able to produce high-quality translations for more creative text types such as literature and whether they are able to generate coherent translations on document level. Our study aimed to investigate these two questions by carrying out a document-level evaluation of the raw NMT output of an entire novel. We translated Agatha Christie's novel The Mysterious Affair at Styles with Google's NMT system from English into Dutch and annotated it in two steps: first all fluency errors, then all accuracy errors. We report on the overall quality, determine the remaining issues, compare the most frequent error types to those in general-domain MT, and investigate whether any accuracy and fluency errors co-occur regularly. Additionally, we assess the inter-annotator agreement on the first chapter of the novel.