An AI-powered voice assistant can handle a refund request over the phone without a human agent. According to a guide published by ElevenLabs, these assistants, which combine speech recognition, a language model, and speech synthesis, are proliferating in corporate customer service. The document details the steps to build one.
In concrete terms, an AI voice assistant is software that understands speech and allows the user to act on connected systems. It combines three components: speech recognition (speech to text), a large language model (LLM, capable of reasoning about text), and speech synthesis (text to speech). Beyond transcription, it can launch an application, retrieve data, or control machines. Consumer assistants like Siri, Alexa, or Google Assistant mainly respond to simple commands, while organizations are developing them for more complex processes. The guide cites insurer Admiral, which designed a voice assistant for millions of customers, capable of answering questions, consulting documents, and offloading repetitive tasks from human agents.
Define the Mission and Choose Your Tools
Before writing a single line of code, the guide recommends defining the agent’s mission, consulting domain experts on user difficulties, and setting measurable indicators. For a sales process, for example, the agent must know how to ask pre-qualification questions. According to the document, ElevenLabs’ SDR (sales development representative) agent, named Dom, achieved a 92% qualification accuracy rate.
The developer must then choose between two paths. Raw APIs, accessible via ElevenAPI, offer total control over the language model, orchestration, telephony, and monitoring, but require building and maintaining everything. The managed platform ElevenAgents bundles models, tools, a knowledge base, and connectors to services like Salesforce, Zendesk, or HubSpot, with testing and analytics functions. The guide suggests first validating the use case in ElevenAgents, then writing code only for layers requiring more control.
Latency, Interruptions, and Verification
Response delay is a central point. The guide recommends staying under 500 ms round-trip and indicates that an agent taking more than 700 ms to respond will seem unnatural. According to ElevenLabs, its Scribe v2 Realtime model transcribes speech in under 150 ms, excluding network and application time, and its v4 Turbo synthesis model shows a latency of about 100 ms for inference alone. Voice activity detection (VAD) relies, according to the document, on deep learning algorithms rather than silence duration alone.
For sensitive operations, the guide provides for an explicit verbal confirmation. If a customer wants to cancel an order, the agent must ask: ‘Are you sure you want to cancel the order?’, then wait for the response. The caller’s identity must also be verified by cross-referencing their number with a customer file and a question only the account holder knows the answer to, validated by a tool and not by the language model. To limit hallucinations, meaning plausible but inaccurate responses, the guide recommends a retrieval-augmented generation (RAG) system, which connects the model to the company’s document base. Finally, when the conversation becomes tense or exceeds the agent’s capabilities, it can transfer the call to a human with the transcription, via a mechanism the document describes as a hot transfer.
Before launch, the guide proposes testing latency, interruptions, turn-taking, accuracy in noisy environments, and security, then tracking after deployment the resolution rate, escalation rate to a human, word error rate, and customer satisfaction.
