# Realtime APIs vs the Cascaded Voice Pipeline A technical look at why realtime speech-to-speech APIs outperform the traditional STT-LLM-TTS pipeline for conversational voice agents: latency, interruption handling, prosody, and architectural simplicity. Source: https://www.voxen.tech/blog/realtime-api-vs-cascaded-pipeline Published: 2026-09-27 Last updated: 2026-09-27 Author: Voxen For most of the last decade, building a voice agent meant assembling three separate systems: automatic speech recognition (ASR) to turn audio into text, a language model to decide what to say, and text-to-speech (TTS) to turn that decision back into audio. Call this the **cascaded pipeline**. It got voice AI to where it is today, and it's still the default in a lot of production systems. **Realtime speech-to-speech APIs** replace all three stages with a single model that listens to audio and responds with audio directly, over one persistent connection. No intermediate transcript, no handoff between separately-trained systems, no waiting for one stage to fully finish before the next one starts. > **The one-line version** > > A cascaded pipeline processes a call in three sequential steps. A realtime API processes it as one > continuous stream. That single architectural change is why realtime feels less like talking to a > phone tree and more like talking to a person. ## The quick answer If you're building a conversational voice agent and can choose your architecture, realtime speech-to-speech is the stronger foundation. It cuts turn-taking latency, handles interruptions the way a human listener does, picks up on tone and pacing that a transcript throws away, and replaces three vendor integrations with one connection. The rest of this post goes through why, stage by stage. ## Two ways to build a voice agent **The cascaded pipeline** runs a caller's audio through three independent models, each trained separately and each unaware of the others: 1. **ASR** converts speech to text. 2. **An LLM** reads that text and generates a reply as text. 3. **TTS** converts the reply back into audio. Each stage has to fully finish before the next one starts. The LLM can't begin composing a reply until ASR has finished transcribing, and TTS can't start speaking until the LLM has finished generating. **A realtime API** collapses this into one model that takes streaming audio in and produces streaming audio out. There's no discrete "transcription complete" event blocking the next stage — the model is reasoning about the conversation continuously, the same way a person listening to you doesn't wait for you to finish a sentence before starting to formulate a response. That single design difference is the root of nearly every advantage below. ## Latency and turn-taking In a cascaded pipeline, latency is additive: ASR latency, plus LLM time-to-first-token, plus TTS synthesis time, plus the network hops between each vendor if they're not co-located. Every stage you add is another stage the caller has to wait through before hearing a single word back. A realtime API collapses that chain into one hop. The model can start generating audio output while it's still processing the tail end of the caller's input, much like a person starts forming a response before the other person has finished talking. The practical result is a response that begins closer to the moment the caller stops speaking, which is the single biggest factor in whether a voice interaction feels like a conversation or a series of walkie-talkie exchanges. ## Interruptions and natural conversation Real conversations are not strictly turn-based. People interrupt, backchannel ("mm-hm," "right," "got it") while the other person is still talking, and talk over each other briefly without derailing the exchange. A cascaded pipeline has to bolt this on: some separate voice-activity-detection component has to notice the caller started talking, signal the orchestration layer to cancel whatever TTS audio is currently playing, and then restart the whole ASR → LLM → TTS chain from scratch. A realtime model handles this as a native property of the architecture rather than a special case. Because it's processing audio continuously rather than waiting for a text handoff, it can respond to an interruption as part of the same ongoing stream — stopping, listening, and picking the conversation back up without the visible seam of "cancel the old pipeline run, start a new one." ## Hearing more than words Transcription is lossy by design — it captures words and discards nearly everything else about how they were said. A cascaded pipeline's LLM stage only ever sees text, so it has no access to pitch, pace, hesitation, emphasis, or tone. A caller who says "yeah... I guess that's fine" with a flat, reluctant tone produces the identical transcript as someone who says it warmly. The words are the same; the meaning isn't. A realtime model reasons over the actual audio, so paralinguistic signal — tone, urgency, hesitation, emphasis — is available input, not something thrown away three stages before the model gets to think about it. That's information a text-only pipeline structurally cannot recover, no matter how good the LLM stage is. ## A simpler system to run A cascaded pipeline is, at minimum, three vendor integrations, three sets of rate limits and quotas, three latency budgets to monitor, and three places a call can degrade or fail. Coordinating cancellation, retries, and failover across all three — while keeping the conversation state consistent — is its own significant engineering surface. A realtime API is a single persistent connection. There's one thing to monitor, one vendor relationship, one latency budget, and one failure mode to design around instead of three interacting ones. For a team building and operating a voice agent, that's meaningfully less orchestration code, fewer integration points that can drift out of sync, and a smaller surface area for the kind of subtle bugs that only show up when three independently-trained systems disagree about what just happened in a call. ## Taking actions mid-conversation A useful voice agent doesn't just talk — it checks calendar availability, looks up an order, or books an appointment while the conversation is still happening. Modern realtime APIs support function calling natively within the audio stream: the model can pause to invoke a tool, get a result, and fold that result back into the spoken response, all within the same continuous session it's already using to talk to the caller. There's no need to serialize the conversation out to a separate text-based orchestration layer just to trigger an action and then resynthesize the result back into speech — it happens in the same loop that's already running. ## Where the industry is heading Realtime speech-to-speech is a newer category than the cascaded pipeline, and it shows in how quickly it's moving: major model providers have shipped realtime APIs only in the last couple of years, and the pace of improvement in latency, voice quality, and tool-calling support has been fast. As these models mature, the architectural advantages above compound — lower latency and better interruption handling aren't nice-to-haves for a voice agent, they're the difference between something that feels like a real conversation and something that feels like a phone tree with a nicer voice. ## What to evaluate if you're building on this If you're deciding whether to build a voice agent on a realtime API, the questions worth testing for are the ones that map directly to the advantages above: - How does response latency hold up under real network conditions, not just a demo environment? - How cleanly does the model handle a caller interrupting mid-sentence? - Does it pick up on tone well enough to change its response when a caller sounds frustrated versus casual? - How well does function/tool calling work mid-conversation, and how much latency does invoking a tool add to the turn? - How does it perform in the languages and accents your callers actually use? Testing these with realistic calls — not scripted, clean-audio demos — is the only reliable way to know how a realtime voice agent will actually perform for your business. > **Related reading** > > For how this plays out for a caller who's used to pressing buttons on a phone menu, see [Voice > Agent vs IVR](/blog/voice-agent-vs-ivr). For how to test any voice agent — realtime or otherwise — > before it goes live, see [Test your agent before going > live](/docs/test-your-agent-before-going-live).