1794571413-logo-v2
SIGN IN

Unpacking the Technical Challenges of Full-Duplex Conversation

1750128659-英文logo带背景

Posted at 13時間 ago

Judging from the research trends emerging around Interspeech this year, the development of Duplex technology is shifting from "proving that it can work" to "making it work well".

The Architectural Debate: End-to-End vs. Modular

Full-duplex models generally follow two main architectural approaches. End-to-end models, such as Moshi, jointly model the user's and system's audio streams within a unified architecture, achieving latencies in the range of 200 ms. The other approach is modular that retains the established ASR-LLM-TTS pipeline but processes streaming audio in "micro-turns" to support real-time barge-in.

The advantage of the modular approach is that it can continue to leverage the strong reasoning capabilities of text-based LLMs. However, a major challenge is enabling LLMs to "perceive" paralinguistic information in speech. At the conference, one start-up presented an innovative solution: using "single-token supervision" to turn a frozen LLM into an efficient decision classifier. This allows the model to quickly determine, without decoding, "whether the user has finished speaking" and "what their emotional state is".

Training Strategies: The Interplay of Data and Alignment

The biggest challenge in training full-duplex models is the lack of authentic two-party conversational data. DualTurn research reveals a key finding: the ability to handle conversational interactions comes primarily from pre-training rather than model capacity. Experiments show that, before pre-training, an 8M-parameter LSTM and a 0.5B-parameter LLM perform almost equally well in barge-in detection. However, after dual-channel pre-training is introduced, barge-in detection performance surges by 340%, with more than 99% of the improvement attributable to pre-training rather than model size.

In terms of model optimization, reinforcement learning is increasingly being used to improve conversational interactivity. One study uses GRPO to train models on authentic conversational audio segments containing interruptions and backchannels, achieving improvements across multiple metrics on Full-Duplex-Bench.

Engineering Breakthroughs: Benchmarks, Privacy, and Tool Calling

Tool-calling capabilities remain a major bottleneck to the practical deployment of full-duplex systems. Current commercial full-duplex models achieve task completion rates of only 31%-51% on customer service tasks, far below the 85% achieved by text-based agents. Emerging approaches propose "outsourcing" tool execution to a backend LLM, while the frontend handles only real-time streaming voice interactions. This architecture combines low-latency voice interaction with the powerful capabilities of an intelligent agent.

💡Final Thoughts: Even the Sharpest Algorithms Starve Without Good Data

Whether the approach is end-to-end or modular, and whether the training method is supervised fine-tuning (SFT) or reinforcement learning (RL), the industry ultimately faces the same challenge: a lack of high-quality, authentic, and natural full-duplex conversational data containing rich interaction patterns, including interruptions, overlapping speech, backchannels, emotional cues, and other paralinguistic features.

Synthetic data may provide a temporary solution, but for models to truly "understand" the rhythm and subtle nuances of human conversation, high-quality interaction data from real-world scenarios is essential.

Related Datasets

Datasets Download Rank

ASR-RAMC-BigCCSC: A Chinese Conversational Speech Corpus
Multi-Modal Driver Behaviors Dataset for DMS
ASR-SCCantDuSC: A Scripted Chinese Cantonese (Canton) Daily-use Speech Corpus
ASR-SCSichDiaDuSC: A Scripted Chinese Sichuan Dialect Daily-use Speech Corpus
ASR-CCantCSC: A Chinese Cantonese (Canton) Conversational Speech Corpus
ASR-SCCantCabSC: A Scripted Chinese Cantonese (Canton) Cabin Speech Corpus
ASR-CShhiDiaCSC: A Chinese Shanghai Dialect Conversational Speech Corpus
ASR-EgArbCSC: An Egyptian Arabic Conversational Speech Corpus
ASR-SCShhiDiaDuSC: A Scripted Chinese Shanghai Dialect Daily-use Speech Corpus