On November 5, Boson AI—founded by Mu Li and his team—released Higgs Realtime Duplex, a full-duplex interactive multimodal model. Their blog outlines the model's architecture and design philosophy: it continuously processes audio and visual inputs, generating speech and digital human avatars while simultaneously performing background reasoning, invoking tools, and requesting assistance from larger models when needed.
Model Website: https://www.boson.ai/blog/higgs-realtime-duplex
These capabilities directly address several practical pain points of traditional voice assistants: When a user pauses midway through a sentence, should the system wait or chime in? When a user says "Mhm," are they simply agreeing or preparing to interrupt? If a search is still in progress, how can the system maintain the conversation while adapting to the user's new demands?
Understanding this model requires looking through the evolutionary lens of voice systems—observing how they balance real-time interaction, task execution, and cost, and what demands these shifts place on training data.
I. From Turn-Based Processing to Continuous Decision-Making: What Do 160 Milliseconds Change?
Most traditional voice assistants rely on a cascade architecture: automatic speech recognition (ASR) converts user speech into text, a language model (LM) generates the response, and text-to-speech (TTS) turns it back into audio. Voice activity detection (VAD) and endpointing help the system decide when to start processing and when to cede the floor.
This architecture offers clear divisions of labor, making it easy to swap models, debug errors, and integrate with existing enterprise systems. However, its limitations primarily stem from information handoffs between modules: when only transcribed text is passed between modules, tone, pacing, and ambient sound must be conveyed separately, while multi-stage processing can accumulate latency.Streaming pipelines can overlap these stages to reduce waiting times, but they don't inherently solve information representation or interaction judgment problems.
End-to-end speech models attempt to shorten this pipeline by letting models directly use audio representations to comprehend inputs and generate speech, enabling acoustic preservation and joint optimization. Even so, this raises the bar for training and debugging: when failures occur, isolating the issue to an independent module becomes much harder than in cascade systems.
Crucially, being end-to-end does not automatically mean being full-duplex. Even if a model can directly listen and speak, it may still wait until the user finishes a full speaking turn before formulating a reply.
Higgs Realtime Duplex further alters how interaction time is structured. It refines the traditional turn-based organization of inputs and outputs into continuous 160-millisecond time slices: continuously reading new information, generating output for the next time slice, and factoring silence directly into the processing loop.
Consequently, the model must constantly evaluate: Is the user going to keep talking? Is the current utterance directed at the system? Should it wait, backchannel, or speak up? Note that 160 milliseconds represents the processing granularity; actual response latency is also influenced by compute, buffering, network conditions, and intent recognition.
The leap from cascade to end-to-end focuses on improving information flow, whereas moving to full-duplex further demands that the system learn to control when to take the floor. Evaluating such models requires looking beyond response content to check whether they cut people off, miss responses, or adapt promptly when interrupted. A signature design contribution of Higgs Realtime Duplex in this direction is breaking down rigid, turn-based processing into continuous time-slice perception and generation—enabling the model to continuously assess, prepare responses, and adjust actions even while the user is still speaking.
II. From Waiting for Results to Asynchronous Collaboration: How to Stay Conversational Without Tangling Tasks
Fetching flight data, running searches, or calling complex reasoning models takes time. If an assistant must stall its dialogue until every background task finishes, conversations will be peppered with awkward silences. Asynchronous collaboration allows the assistant to keep receiving user input and responding while waiting for results.
Through engineering orchestration, such systems split responsibilities: a frontend manages the dialogue while a backend handles queries, and a filler phrase like "Let me check that for you" can be spoken before the task executes. However, pre-scripted prompts require fundamentally different capabilities than continuously understanding users and dynamically adjusting tasks.
The design focus of Higgs Realtime Duplex is enabling the frontend interactive model itself to handle reasoning and tool invocation, reserving calls to a larger backend model for more complex queries. Combined with the continuous time-slice mechanism, it can begin preparing responses before the user finishes speaking, or keep listening and talking while waiting for tool outputs. The potential payoff is overlapping some of the computation with the ongoing conversation, shrinking post-utterance waiting periods and reducing repeated calls to larger models.
Yet, starting early also introduces the need for timely correction. For example, if the assistant has already searched for "flights to Shanghai tomorrow," but the user suddenly changes the request to "the day after tomorrow afternoon," the system must update query parameters and prevent stale search results from polluting the current answer. When tools have not yet returned concrete data, the assistant can report progress but must never prematurely confirm pricing or schedules. For operations with real-world consequences—such as booking tickets—query actions must be strictly decoupled from final execution.
Therefore, evaluating this design requires tracking two metrics: whether waiting times are shortened and whether tasks are completed accurately. The model's ability to update conditions when a user changes their mind, correctly discard outdated results, and deliver substantive progress updates during background waits reveals far more about its capabilities than simply its ability to chatter endlessly.
III. From Audio Transcription to Interaction Processes: What Duplex Data Demands
These mechanisms place heavier burdens on training data. Ordinary "audio + text" pairs can teach a model to recognize and express content, but they fail to explain why a particular pause warranted waiting, or why a specific backchannel shouldn't trigger a stop.
Duplex corpora must preserve a shared timeline, recording dual-stream speech, silence, overlaps, and event boundaries. Silence must not be wiped out during preprocessing, as it often encapsulates thinking, waiting, and turn-taking dynamics. Raw multi-track audio and actual playback timestamps should be retained as much as possible to distinguish model behavior from system buffering artifacts.
Furthermore, acoustic phenomena and interaction dynamics should be annotated separately. Overlapping speech can be localized, but determining whether it represents backchanneling, competitive interruption, or task correction requires contextual interpretation. Datasets should also cover "non-response" scenarios, such as overhearing background conversations or talking to oneself, while noting what auditory and visual cues were available to the model at the time.
A deeper requirement is maintaining information boundaries during decision-making. While human annotators reviewing a full recording know how a conversation ultimately ends up, the model at that specific timestamp did not. Key samples can benefit from truncated context reviews: assessing rational behavior based strictly on information available up to that moment, followed by full-process checks. When ambiguity arises, multiple acceptable actions and their governing conditions can be logged.
From the perspective of MagicHub.com, full-duplex models have elevated data quality requirements all the way to continuous interaction tracking. Usable data must clearly detail what the system knew at the time, what actions it took, and why new information prompted adjustments. Only by documenting these processes transparently can developers pinpoint where an interaction failure occurred and systematically improve models and systems.
It is worth noting that WildASR, a benchmark from Boson AI—a company co-founded by Mu Li and others—uses full-duplex dialogue datasets from magichub.com as a key source of authentic, spontaneous human speech.This is vital for uncovering the true robustness gaps of conversational voice systems under real-world conditions.
Paper Link: https://arxiv.org/abs/2603.25727
