The Open ASR Leaderboard, a ranking of automatic speech recognition (ASR) systems, is being enriched with private data through a new partnership with Liva AI. This partnership adds two accents, Egyptian and Nigerian, and allows for a more detailed presentation of results by accent.
A few months earlier, the ranking had already integrated private data: 11 distinct datasets, provided by Appen and DataOceanAI, covering scripted and conversational speech in five accents. Community reactions were numerous, with several highlighting the importance of ‘held-out’ test sets, which allow models to be evaluated on data absent from their training. An additional dataset, dedicated to Indian-accented English, was later added with Voice Arena.
Two New Accents: Egyptian and Nigerian
The partnership with Liva AI brings two new private datasets. The first, Liva AI Conversational EG, represents the Egyptian accent with 1.34 hours of recordings. The second, Liva AI Conversational NG, covers the Nigerian accent with 1.55 hours. Both involve conversational and spontaneous speech, with punctuated transcription, in uppercase and lowercase, that retains disfluencies, i.e., the hesitations, repetitions, and self-corrections typical of spoken language.
According to Hugging Face, this addition is part of an ongoing effort to better represent accents and languages in evaluations and to strengthen confidence in results. The new private data should reduce the risk of ‘benchmaxxing’, i.e., optimizing a model for known tests, while expanding the number of data providers.
A Distribution Based on Kachru’s Three Circles
With these new accents, the ranking can present a finer breakdown, based on Kachru’s three circles, a model that classifies the uses of English worldwide. The idea was borrowed from Hume AI’s ranking page, which Hugging Face explicitly thanks.
The Open ASR Leaderboard will now display two breakdowns. The first contrasts American accents with non-American accents, represented by the UK, Australia, Canada, India, Nigeria, and Egypt. The second distinguishes the inner circle, which groups the US, UK, Australia, and Canada, from the outer circle and the expanding circle, which include India, Nigeria, and Egypt. Hugging Face indicates it will separate these last two circles as new accents are added.
Who is Liva AI
Liva AI collects multimodal interaction data through consumer products used by real people. Because the company controls both the collection environment and the distribution of its products, it can, according to its presentation, design products based on model shortcomings, across different domains, sectors, languages, modalities, and interaction types. Hugging Face’s blog specifies that this data reflects natural, context-rich behavior born from real product usage, not artificially prompted or task-specific interactions.
