Players can now approach a character in a Unity game and talk to them out loud. The new ElevenAgents SDK (software development kit) for Unity connects a scene to an ElevenLabs voice agent, which listens to the player via the microphone, speaks in its voice, and manages the session via the Conversation API. The same code works in the Unity editor, on desktop builds, and in WebGL, i.e., in a browser, while iOS, Android, and XR versions are in beta.
- Players talk out loud to Unity characters through the ElevenAgents SDK.
- The session opens when the player enters a detection zone around the character.
- The agent’s voice can come from the character itself, and its volume can drive animations.
- Authentication must be re-enabled before production, and the API key must never appear in a build.
Starting a conversation
A session requires the ID of an agent created in the ElevenLabs dashboard, which contains the prompt, i.e., the instructions given to the agent, as well as the voice. In the GettingStarted example provided with the SDK, an invisible detection zone placed around the character opens the session when the player enters and closes it when they leave. The StartSessionAsync method only returns once the connection is open and the audio devices are ready, and it selects the transport mode suitable for the target platform. EndSession does nothing if the session is already ended.
Characters that speak, listen, and act
By default, the agent’s voice is played in 2D. By passing its own AudioSource, Unity’s audio component, as OutputAudioSource, the voice originates from the character itself. The GetOutputVolume() method returns the agent’s current sound level, between 0 and 1, on all platforms. The example uses this to make a cube bounce while it speaks, and the same principle can drive a mouth animation.
Dynamic variables fill slots like {{mood}} in the prompt and the agent’s first message, so each character can send its own values at the start of its session. These values can be strings, numbers, or booleans. In the example, a yellow cube and a red cube, whose moods differ, share the same agent. For more significant changes, Overrides replace part of the agent’s configuration for a session, such as the first message or the voice. Each must first be enabled in the agent’s security settings.
The SendContextualUpdate method informs the agent of game events during a conversation, without interrupting it or adding a turn of speech that the player did not utter. Client tools, on the other hand, allow the agent to trigger game code. The tool is defined in the agent’s settings, then a handler with the same name is registered in the game. The SDK deserializes the parameters with Newtonsoft Json and returns to the agent what the handler returns. The handler can return an Awaitable for long tasks and throw a ClientToolException to explain why an action failed.
Events like UserTranscriptReceived, AgentResponded, ModeChanged, or Interrupted are triggered on Unity’s main thread, allowing the interface to be updated directly. SendUserMessage allows sending text, SetMicMuted mutes the microphone, and SetVolume adjusts the volume. Changing the input or output device during a session is also possible.
Platforms and getting started
On iOS and Android, the player must authorize microphone access before starting a session. In WebGL, the Use WebAssembly.Table option must be enabled in the Publishing Settings, otherwise the SDK causes the compilation to fail with instructions. Play mode does not work when WebGL is the active target, so testing must be done on a desktop target. The voice then goes through the Web Audio API, and some AudioSource settings, such as mixer routing or the doppler effect, are not applied. WebRTC sessions ignore OutputAudioSource.
For an initial test, simply create an agent, disable Authentication in its security settings, then install the package from the Git URL indicated in the documentation, pinning version v0.1.0. Then import the GettingStarted example, paste the agent’s ID into it, and launch the game. Before any production deployment, Authentication must be re-enabled: the developer’s server then requests a short-lived signed URL, passed to the game as SignedUrl. The ElevenLabs API key must never appear in a build. The tutorial lasts about 10 minutes and ends with a cube greeting the player. Bugs and feature requests should be reported in the project’s GitHub issues.
