NVIDIA Open-Sources PersonaPlex for Real-Time Full-Duplex AI Conversations

NVIDIA has released PersonaPlex, an open-source speech-to-speech conversational model designed for real-time, full-duplex interactions. This system allows for simultaneous listening and speaking, supporting interruptions, natural interjections, and overlapping dialogue.

PersonaPlex enables users to define the "persona" and voice style of a conversational agent through text prompts and audio samples. Built on the Moshi architecture, the model is trained using both synthetic and real conversational data. It generates low-latency, natural, and fluent voice responses while maintaining character consistency.

Three diverse digital avatars representing different AI personalities: a teacher, customer service, and casual conversationalist.

Customizable AI Personalities

Unlike earlier voice systems often limited to a single voice and personality, PersonaPlex offers flexibility in role definition. Users can configure the AI to act as a teacher, a customer service representative, or a casual conversational partner. The system can also mimic specific speaking voices and intonations based on short audio inputs. However, it does not currently support real-time translation or deployment on mobile devices.

A software developer working on multiple computer screens, displaying lines of code.

Technical Implementation

To use PersonaPlex, developers must install the Opus audio codec development library and then clone and install the Moshi repository. Additional steps are required for GPUs based on the Blackwell architecture. Users also need to accept the PersonaPlex model license on Hugging Face and set their Hugging Face Token.

The model can be run as a server for real-time interaction, accessible via a web user interface. A CPU offload option is available for environments with insufficient GPU memory. PersonaPlex also supports offline evaluation, allowing processing of input WAV files to generate corresponding output WAV files and JSON text.

Server rack with glowing lights and overlaid speech waveforms, symbolizing real-time AI interaction.

Voice Options and Prompting

PersonaPlex includes pre-set voice embeddings categorized as "Natural" (NAT) for conversational use and "Variety" (VAR) for more diverse voices. These include multiple male and female options within each category.

The model's training data includes synthetic conversations with fixed assistant roles and various customer service scenarios, alongside real conversational data. Example prompts demonstrate how to define roles such as a wise teacher, a waste management representative, a restaurant employee, or a drone rental agent.

For casual conversations, the model is trained on real dialogues from the Fisher English Corpus, using LLM-annotated prompts. Users can employ general prompts like "You enjoy having a good conversation" or more specific characterizations, such as "You are David Green, a former baker now living in Boston."

Abstract neural network with glowing nodes and overlaid speech patterns, symbolizing AI generalization.

Generalization Capabilities

PersonaPlex leverages the generalization capabilities of the underlying Helium LLM, which underpins the Moshi architecture. This allows the model to produce relevant responses even when presented with prompts outside its specific training distribution. An example prompt available in the web UI demonstrates a technical discussion about fixing a reactor core on a spaceship, with the AI acting as an astronaut named Alex.

The project's code is available on GitHub.