From 8 seconds to 1.5

The first version of our voice agent worked. You spoke, it listened, it answered. It just took about eight seconds to answer, and eight seconds of silence on a phone feels like the app has frozen.

Waiting for every step

The pipeline was three steps in a row: speech to text, a call to the language model, then text to speech. Each step waited for the one before it to finish completely. The user's sentence had to be fully transcribed before the model saw it, and the model's whole reply had to exist before we started speaking it.

Streaming everything

The fix was to stop waiting. I switched all three steps to streaming: partial transcripts went to the model as they arrived, and the model's words went to the voice as soon as it produced them. The steps now overlap instead of queueing.

Then the voice sounded like a robot

Faster was not enough. We were using the phone's built-in voices, and people noticed. Moving to ElevenLabs and PlayHT made the agent sound human, but it added latency back, so I had to stream that step too.

Betting on a beta

When the OpenAI Realtime API came out, it handled speech in and speech out in one connection. We moved to it while it was still in beta and ran on it for months. It cut latency a lot, and it brought its own problems: background noise, conversations that switch languages halfway, and getting the model to stick to its instructions.

Most of the work was not one big fix. It was removing one wait at a time.

We ended at about 1.5 seconds from the user stopping to the agent starting to speak.