Repository navigation
💡 Full-Duplex Voice Architecture: Simultaneous Listen-and-Speak Pipeline as GPT-Live Competitive Response #522
Replies: 1 comment
Weekly UpdateWhat ChangedMainstream products now combine avatars with real-time, interruptible voice, on the Mac.
What this means for TalkTerm: "an animated avatar you can talk over" is no longer a differentiator on its own; Meta now ships it free on the same desktop OS. TalkTerm's remaining edge is agentic work on local files with confirmations, a combination Muse doesn't claim. That makes the turn-taking and barge-in quality in this idea, together with #575 (warm start) and #493 (voice steering during execution), more important for competitive parity. Updated Assessment
RecommendationAdvance, in order: land #575 (warm start and latency telemetry) first so the voice loop is measurable, then use the existing |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Rebuild TalkTerm's voice pipeline around a full-duplex architecture that can listen and speak simultaneously, matching the interaction quality GPT-Live established in July 2026. Instead of strict turn-taking, the avatar can produce backchannel acknowledgments ("mhmm", "got it") while the user speaks, handle natural interruptions without conversation resets, and overlap speech-end detection with response generation for sub-200ms perceived latency.
Market Signal
OpenAI shipped GPT-Live on July 23, 2026 with full-duplex voice on the ChatGPT desktop app (Mac & Windows). GPT-Live uses a full-duplex architecture that listens and speaks simultaneously, supports backchannel acknowledgments, and treats interruption as native behavior rather than an error. This sets a new competitive bar for conversational AI desktop apps — half-duplex turn-taking now feels dated. GPT-Live-1 shipped at $0.05/minute with ~200ms practical latency on September 10, 2026.
User Signal
Existing ideas #502 (Intelligent Turn-Taking) and #435 (Conversational Backchanneling) identify the need for more natural voice interaction, but neither proposes the full architectural shift to duplex audio that GPT-Live has now normalized. Non-technical users especially notice conversation unnaturalness because they benchmark against human conversation, not against prior AI tools.
Technical Opportunity
TalkTerm's architecture separates STT/TTS behind abstraction interfaces (
SpeechToText,TextToSpeechports inshared/types/ports/), enabling pipeline replacement without UI changes. The Electron main process can manage a full-duplex audio state machine that coordinates microphone capture, STT streaming, TTS playback, and agent response generation as parallel tracks. Rive's multi-layer state machine can drive "listening while speaking" avatar animations concurrently via additive layers. Web Audio API provides the mixing layer in the renderer process.Assessment
Adversarial Review
Strongest objection: Full-duplex is architecturally complex — it requires separating audio transport from agent inference, managing concurrent audio streams, and handling the edge case explosion of both parties speaking simultaneously. A single developer may burn weeks on audio engineering that doesn't ship user-visible features.
Rebuttal: TalkTerm doesn't need to build a full-duplex speech model (like GPT-Live) — it needs a full-duplex audio pipeline that coordinates existing components (STT, TTS, agent) in parallel rather than sequentially. Phase 1 can be backchannel-only (avatar produces "mhmm" during user speech) without full overlap, delivering 80% of the perceived improvement at 20% of the complexity. The abstraction layer is already designed for provider swapping.
Suggested Next Step
Prototype a backchannel-capable audio state machine in the main process that can play short TTS acknowledgment clips while STT capture remains active, using Web Audio API mixing in the renderer.
All reactions