Sentence-by-sentence speech cut an AI agent’s first-audio delay in AWS tests

A voice agent can have the start of an answer ready while its speech service waits for the rest. AWS’s 22 October engineering demonstration removes that wait by sending each completed sentence to text-to-speech as the language model continues writing. The change overlaps two stages of the response rather than speeding up the language model itself.[1]

The implementation uses Kokoro-82M on a SageMaker AI endpoint, with a Strands agent backed by Bedrock and deployed through AgentCore. AWS compared incremental speech with a batch path that waited for the full text, using the same model, endpoint and prompt in the end-to-end comparison.[1]

Three measurements, with separate scopes

In the isolated test, the service received either the whole four-sentence block or only its first sentence. Mean time to the first audio byte fell by 172.6 milliseconds. In the full agent test, the opportunity for improvement was larger because synthesis could overlap with the time the language model spent generating later sentences.[1]

Kokoro’s 82-million-parameter model ran in a custom container on an ml.g5.xlarge GPU endpoint. Two concurrent tasks handled text and audio: one fed completed sentences into synthesis, while the other passed audio chunks back as they arrived. A shared queue preserved event order without requiring the later text to be complete.[1]

Binary audio and browser events serve different legs

The SageMaker synthesis connection can carry binary audio alongside control messages. That avoids the roughly one-third size overhead of representing binary audio as base64 text. The browser-facing AgentCore response still uses JSON-compatible Server-Sent Events and base64 audio. Both batch and incremental modes pay that same browser-leg cost, so the reported comparison isolates when synthesis starts.[1]

Sentence boundaries and buffering remain practical constraints. Very long first sentences delay the point at which speech can start, and interruptions require a way to stop queued audio. Those behaviours need testing in the intended application alongside the headline first-byte timing.[1]

The demonstration identifies a specific avoidable pause: synthesis waiting for text it does not yet need. Letting the first sentence travel ahead of the rest produced the measured gain. That scheduling choice is reproducible, while the size of the benefit depends on the response and deployment.[1]

Report an error or suggest a correction