Speech synthesis, or text to speech (TTS), transforms written text into spoken audio. According to the ElevenLabs article, the system first analyzes the text to determine how each element should be pronounced, then an artificial intelligence model generates the sound.
- Speech synthesis converts written text into audio through a frontend and a backend.
- The frontend splits text into tokens and interprets symbols, and it relies on context to resolve ambiguities.
- Neural engines produce more natural speech than older formant and concatenative methods.
- Speech synthesis serves narration, accessibility, and customer support, and pronunciation can be controlled with dictionaries and phonetic tags.
How speech synthesis works
The process occurs in two main phases. The first, called the frontend, cleans the text, breaks it down into tokens (language units, which can be a word, part of a word, or a punctuation mark) and converts it into mathematical representations, called vectors, which the system can manipulate. The second, called the backend, transforms this data into waveforms that model speech.
The engine must also interpret symbols. It thus reads “$11” as “eleven dollars.” It must also manage prosody, meaning the pitch and rhythm of the voice, which give speech its natural character. It relies on context for this, notably for homographs, words that are spelled the same but are not always pronounced the same way.
The article follows a sentence, “Mr. Ríos paid $11 on 10/06.”, through the process. The engine expands “Mr.” to “Mister”, reads “$11” as “eleven dollars” and decides whether “10/06” means “October sixth” or “the tenth of June”, depending on language and context. It then places emphasis on “eleven dollars”, adds a short pause before the date, and lowers the voice pitch at the end of the sentence. The acoustic model then produces a mel-spectrogram, a representation of sound frequencies and energy over time, which the vocoder converts into an audio wave. An upstream error, like a misread date, therefore appears in the final sound.
Why neural voice sounds more human
Neural engines, the most recent, use a deep neural network, trained on large volumes of voice recordings paired with their text. They excel at producing natural and intelligible speech, two criteria engineers use to evaluate these engines.
They succeed two generations of technologies. Formant synthesis, dominant in the 1970s and 1980s, generated sounds modeled from mathematical rules. It was intelligible, but monotonous and robotic. Concatenative synthesis, used from the 1990s to the 2010s, assembled small samples of human voice extracted from a vast corpus. More natural, it could however seem choppy.
The article also distinguishes speech synthesis from “AI voice generators”, a marketing term used to describe a broader set of technologies, including voice design, voice cloning, and voice style transfer. Speech synthesis is one part of it.
Uses and pronunciation control
Speech synthesis is used for narration, voice-overs, dubbing, audiobooks, and video game dialogue. The article cites the case of TAP, which used ElevenLabs TTS to produce educational content for schools in India. It is also used for accessibility, notably with screen readers, reading assistance for dyslexic people, and language learning. In customer support, voice agents combine speech recognition, LLM (large language model), and connected tools. According to the source, Klarna reduced its resolution time by 10x thanks to ElevenAgents.
To correct pronunciation errors, particularly for proper names or jargon, the ElevenAPI offers pronunciation dictionaries in PLS format, which associate spellings with phonemes, as well as phonetic tags in IPA notation (International Phonetic Alphabet) directly in the text. Developers can also connect their application via an API, by sending text, a voice ID, and a model ID. According to the article, latency can be reduced to around 100 ms before the first sound for conversational agents.
