A simple fix for LLM tail latency
Zhixuan Lai | June 10, 2026
When LLM responses are too slow for your realtime use case, you may be tempted to pay double the cost for a faster service tier. Anthropic’s Priority tier, OpenAI’s priority processing, Gemini’s priority inference, whatever your LLM provider calls it. There’s a simpler solution: send every request twice and take the faster response.
Why tail latency matters for voice agents
Our voice agent at HOAi answers phone calls. Every turn in a conversation makes an LLM request. Most responses come back within 1.5 seconds, but occasionally one takes 10 to 20 seconds. On a phone call, that’s 10 seconds of awkward silence, and after enough silence, the caller hangs up on our agent.
This happens more often than you’d think. A typical phone call has 20 to 30 turns. If 1% of LLM requests are catastrophically slow, a 25-turn call has roughly a 22% chance of hitting a long silence.
Priority tier vs. sending each request twice
We had two options.
- Upgrade to OpenAI’s priority tier and pay 2x cost per token for faster, more consistent responses.
- Stay on standard tier, but send every request twice and take the faster response.
We replayed 50 real production requests against both setups and tracked two metrics: time to first token (when the agent starts speaking) and time to complete response (when it can act on tool calls).
Time to first token:
| Priority tier | Standard tier, sent twice | |
|---|---|---|
| median | 0.61s | 0.58s |
| p95 | 1.04s | 0.68s |
| p99 | 4.2s | 1.2s |
Time to complete response:
| Priority tier | Standard tier, sent twice | |
|---|---|---|
| median | 1.35s | 1.35s |
| p95 | 3.4s | 2.0s |
| p99 | 9.8s | 3.5s |
| worst | 9.8s | 3.5s |
Sending the request twice clearly outperformed the priority tier. Worst-case time to complete response dropped from 9.8s to 3.5s. Worst-case time to first token dropped from 4.2s to 1.2s. Even the median matched the priority tier exactly, despite the standard tier being slower per individual request.
This works when slow responses are rare and independent. Sending the request twice makes it unlikely both copies are slow on the same turn. This significantly reduced the 10-second silences our callers were experiencing.
The takeaway
If you are building a realtime interactive LLM product, before you pay for the faster service tier, benchmark it against sending the request twice. You may get better latency at the same cost.
