Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Kreutzer, Julia; Caswell, Isaac; Wang, Lisa; Wahab, Ahsan; van Esch, Daan; Ulzii-Orshikh, Nasanbayar; Auguste Tapo, Allahsera; Subramani, Nishant; Sokolov, Artem; Sikasote, Claytone; Setyawan, Monang; Sarin, Supheakmungkol; Samb, Sokhar; Sagot, Benoît; E. Rivera, Clara; Rios, Annette; Papadimitriou, Isabel; Osei, Salomey; Ortiz Suárez, Pedro; Orife, Iroro; Ogueji, Kelechi; Niyongabo Rubungo, Andre; Nguyen, Toan; Müller, Mathias; Müller, André; Hassan Muhammad, Shamsuddeen; Sakib Muhammad, N; Mnyakeni, Ayanda; Mirzakhalov, Jamshidbek; Matangira, Tapiwanashe; Leong, Colin; Lawson, Nze; Kudugunta, Sneha; Jernite, Yacine; Jenny, Mathias; Fırat, Orhan; F. P. Dossou, Bonaventure; Dlamini, Sakhile; de Silva, Nisansa; Çabuk Ballı, Sakine; Biderman, Stella; Battisti, Alessia; Baruwa, Ahmed; Bapna, Ankur; Baljekar, Pallavi; Abebe Azime, Israel; Awokoya, Ayodele; Ataman, Duygu; Ahia, Orevaoghene; Ahia, Oghenefego; Agrawal, Sweta; Adeyemi, Mofetoluwa

doi:10.1162/tacl_a_00447

Public

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Published in Transactions of the Association for Computational Linguistics • Jan 1, 2022

Authors:

Julia Kreutzer

Isaac Caswell

Lisa Wang

Abstract

Abstract With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, Web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with ...

Subject

Computer science

USable

Audit

Research Assistant

AI chat, annotations, notes & similar papers

Finding related papers...

Discussions

(0)

No comments yet

Be the first to share your thoughts!