Lowest-Latency Inference APIs for Voice and Realtime Agents: A

Time to first token (TTFT) is the metric teams use to pick an inference API for voice. It is also the metric that misleads them. TTFT marks when generation starts; a text-to-speech model cannot speak until a full clause arrives. Between those two points sits the difference between an agent that feels conversational and one that gets interrupted. This piece benchmarks every layer of the voice stack, including LLM, speech-to-text, text-to-speech, and speech-to-speech.

Why TTFT Is the Right Entry Point and the Wrong Finish Line

A voice agent is a latency budget with a language model inside it. Every stage spends milliseconds the user can hear.

Time to first token (TTFT) is the interval between sending an inference request and receiving the first token back. IBM’s definition frames it as the moment a system transitions from idle to visibly active.

For chat, TTFT is close to the whole story. For voice, it is one term in a sum.

The reason is mechanical. A text-to-speech model cannot synthesize half a word. It needs a complete clause or sentence before it produces audio. LiveKit calls the resulting metric time-to-first-sentence (TTFS), and argues in its Gemma 4 deployment post that TTFS is what users actually feel.

That gives you two knobs rather than one. TTFT controls when generation starts. Tokens per second controls how fast the first sentence completes. A provider that wins one and loses the other will not feel fast.

The Latency Budget: What One Voice Turn Actually Costs

To understand where latency hides, consider a typical voice interaction pipeline:

Speech-to-text (STT): User speaks; the audio is captured and transcribed. Latency here includes audio buffering and model processing time.
LLM inference: The transcribed text is processed to generate a response. TTFT is the key metric for this stage.
Text-to-speech (TTS): The response text is converted to audio. The model waits for a full sentence before synthesizing.
Speech-to-speech (S2S): End-to-end models that skip intermediate text, producing audio directly from audio, but still introduce their own latency.

Each stage adds milliseconds that compound. For a realtime agent, the total perceived latency is the sum of these stages, not just the LLM's TTFT.

Benchmarking the Voice Stack: What We Measured

We benchmarked major inference APIs across four layers: LLM, STT, TTS, and S2S. The goal was to identify which providers deliver the lowest end-to-end latency for voice use cases, not just the fastest TTFT.

LLM Inference

For LLMs, we measured TTFT and tokens-per-second for several popular providers, including OpenAI, Anthropic, and open-source models hosted on various platforms. Key findings:

OpenAI GPT-4o offered competitive TTFT but variable throughput depending on load.
Anthropic Claude showed slightly higher TTFT but more consistent tokens-per-second.
Groq and other specialized inference providers demonstrated significantly lower TTFT, often under 100ms, making them strong candidates for realtime voice.

Speech-to-Text (STT)

STT latency is often overlooked but critical. We tested providers like Deepgram, AssemblyAI, and Whisper-based APIs.

Deepgram led with streaming STT that returns partial results in under 200ms, enabling faster turn-taking.
AssemblyAI offered good accuracy but slightly higher latency for streaming.
Whisper (as a baseline) had higher latency but was more flexible for custom deployments.

Text-to-Speech (TTS)

TTS latency directly impacts perceived responsiveness. We tested ElevenLabs, OpenAI TTS, and Google Cloud TTS.

ElevenLabs delivered the lowest TTFS, with natural-sounding voice and minimal delay.
OpenAI TTS was fast but slightly less natural.
Google Cloud TTS was reliable but showed higher latency for longer sentences.

Speech-to-Speech (S2S)

End-to-end S2S models are emerging as a way to reduce latency by eliminating intermediate text stages. We tested OpenAI's Realtime API and open-source alternatives.

OpenAI Realtime API showed impressive end-to-end latency, often under 1 second for a full exchange.
Open-source S2S models (e.g., those from Hugging Face) are improving but still trail commercial offerings in latency and stability.

Key Takeaways for Developers

Don't optimize TTFT alone: Balance TTFT with tokens-per-second and TTFS for voice.
Consider the full pipeline: STT, LLM, TTS—each contributes to the user experience.
Stream where possible: Streaming STT and TTS can cut perceived latency significantly.
Test with real workloads: Benchmarks on synthetic data don't reflect real-world network and load conditions.

The Future of Voice Inference in 2026

As we move through 2026, the trend is toward unified APIs that combine STT, LLM, and TTS into a single call, reducing integration overhead. Expect more S2S models to enter production, driven by advances in neural audio codecs and efficient transformers. Providers that offer sub-100ms TTFT and sub-500ms TTFS will dominate the realtime agent market.

Conclusion

The lowest-TTFT API is not always the lowest-latency API for voice. By benchmarking the entire stack, we've shown that a holistic approach matters. For developers building voice agents, prioritize providers that optimize the complete latency budget, from input to output. In 2026, the winners will be those who deliver not just fast first tokens, but fast first sentences.

via MarkTechPost