Recently, Adventists AI officially open-sourced Sophon-Realtime v1, a full-duplex speech model, making its premiere on the ModelScope community. The release quickly sparked widespread discussion around its expressiveness, multilingual capabilities, and deployment threshold.
To address the most frequently discussed topics, the Magichub.com open-source community sat down with Jin Jie, Founder of Adventists AI, for an exclusive Q&A. Read on to discover the key technical insights the community is buzzing about!
Demo Preview
Jin Jie:
Hi everyone, greetings to all our developer friends!
On November 8, Adventists AI open-sourced Sophon-Realtime v1, a full-duplex speech-to-speech model. It listens the whole time you talk and hears hesitation, stress and sarcasm. When it answers, it laughs or sighs when the moment calls for it. Code and weights are Apache-2.0, and it runs on a single 80 GB GPU.
We’ve had a lot of feedback since the launch. This piece covers what the launch post didn’t have room for: why the voice feels human, which languages it speaks, and the releases coming in the next few weeks.
1. It hears your tone, and answers with one
Most voice assistants turn speech into text first, and the tone is lost at that step. Sophon-Realtime’s brain listens to the audio directly. On public paralinguistic benchmarks (LISTEN, StressTest, MSPB) it scores 48.6 overall.
While it writes a reply, it inserts sound tags for laughs, sighs, breaths and stressed words. The voice performs them instead of reading them aloud. On NVV-SuperBench, which scores non-verbal vocalizations, the Tag build scores 0.86 in Chinese and 0.80 in English.
The line people shared most after the launch: “an open model that hears your sarcasm and laughs back.”
2. Not just Chinese and English
Our formal benchmarks are in Chinese and English, but the model isn’t limited to them. Early testers report that French and Japanese work well. Japanese is a language we have invested in from the start, and the next releases will push it further.
3. Faster: the new end-to-end build reaches about 0.2 s
Sophon-Realtime comes in three builds that share one brain. Tag is the most expressive, with first audio in about 0.6 s. Flash reaches first audio in 0.31 s and serves 40 conversations per GPU. There is also an end-to-end (E2E) research build.
The upcoming E2E build shrinks the speaking module from 32B to 220M parameters. First audio for a single conversation drops to about 0.2 s, more than twice as fast as the previous build’s 0.51 s.
The E2E build also differs from Tag in where its expressiveness comes from. In Tag, the brain writes tags for laughs and sighs into the text, and the voice performs them. The E2E build doesn’t rely on tags: the speaking module learns tone, laughter and pauses end to end, directly from real conversational recordings, so they emerge naturally from context instead of being scripted.
4. Smaller and smarter: version 1.1 next week
Version 1.1 moves to a 27B backbone: smaller than v1’s 32B, and smarter. The speaking module is decoupled from the brain, so changing the backbone doesn’t affect the voice.
5. Easier to deploy: consumer GPUs
- Quantized build: ships with the new E2E build, targeting a single 24 GB consumer GPU.
- Available today: one 80 GB GPU and two commands, pip install sophon-realtime and sophon-realtime serve. It speaks the OpenAI Realtime protocol, so LiveKit and Pipecat connect by changing the base URL.
6. On your Mac: Apple MLX build this month
Many developers don’t have an 80 GB NVIDIA GPU, but they do have a Mac with plenty of memory. On day two after the launch, a developer opened a GitHub issue asking to run Sophon-Realtime locally on an M5 Max with 128 GB of unified memory. It is one of the questions we’ve heard most since the launch.
We plan to release an Apple MLX build later this month:
- No CUDA needed: runs locally on Apple Silicon Macs, no NVIDIA GPU required.
- Your voice stays on your machine: the whole conversation runs on your own computer, which suits people and teams who care about privacy.
- Requirements: the 4-bit build needs 32 GB or more of unified memory; 64 GB and 128 GB machines can run higher precision.
- Built with the community: we’ll invite community developers to test the first builds on different Macs, and publish latency and quality numbers together.
7. Good models come from good data
A real-time conversation model has to learn more than what to say. It has to learn how people talk: pauses, interruptions, back-channels, the rise and fall of tone. That only comes from real conversational recordings. Sophon-Realtime was trained with MagicData’s Chinese conversational speech data. Its natural recordings and careful annotation helped the model learn to talk like a person.
Try it
- Online demo (ModelScope Studio): https://www.modelscope.ai/studios/AdventistsAI/Sophon-Realtime
- Code: https://github.com/adventists-ai/sophon-realtime
- Model: https://huggingface.co/adventists-ai/Sophon-Realtime-v1
Dataset Link
- Multi-stream Spontaneous Conversation Training Datasets_Chinese : https://magichub.com/datasets/multi-stream-spontaneous-conversation-training-datasets_chinese/
- Multi-stream Spontaneous Conversation Training Datasets_English : https://magichub.com/datasets/multi-stream-spontaneous-conversation-training-datasets_english/
- Japanese Duplex Conversation Training Dataset : https://magichub.com/datasets/japanese-duplex-conversation-training-dataset/
