When evaluating Large Language Model (LLM) API performance, latency is one of the most critical operational metrics. Yet many engineering teams and technical decision-makers fall into the trap of expressing latency as a single average. While average response time can already provide a skewed picture in traditional web applications, the arithmetic mean is downright misleading for LLM infrastructure.
LLM interactions do not follow a classic normal distribution (the familiar bell curve), but rather a heavily skewed distribution with an extremely long tail (a so-called heavy-tailed distribution). Making decisions based on averages ignores the frustrating experience of the 5 to 10 percent of users who encounter unacceptably slow requests. In this guide, we cover why the latency distribution in LLMs is so skewed, how to interpret percentiles (p50, p90, p95, p99), how to properly normalize measurements for output length, and which pitfalls exist when aggregating data and establishing a Service Level Objective (SLO).
The arithmetic mean is calculated by dividing the sum of all measured response times by the total number of measurements. This metric is extremely sensitive to outliers. When nineteen requests are completed within 500 milliseconds, but the twentieth takes 30 seconds due to a cold start mechanism or retry, the average across all twenty requests instantly jumps to nearly 2 seconds.
However, that 2-second average fails to tell you two things:
For a broader overview of setting up performance benchmarks, you can consult the guide on measuring speed in LLMs , but the root cause of this specific latency behavior lies in the architecture of AI cloud providers.
The response time of an LLM endpoint is not determined solely by the compute required for a prompt, but by a chain of dynamic factors at the provider level:
To properly understand the distribution of response times, we use percentiles. A percentile is a threshold indicating what percentage of measurements falls at or below that value when all measurements are sorted in ascending order.
| Percentile | Practical meaning | Operational focus |
|---|---|---|
| p50 (Median) | 50% of requests were faster than this value, and 50% were slower. This represents the 'typical' experience. | Insight into the overall efficiency of the infrastructure under normal conditions. |
| p90 | 90% of requests were faster than this value; 10% experienced higher latency. | First indication of building traffic or minor capacity constraints. |
| p95 | 95% of requests fell within this time. 1 in 20 requests was slower. | The standard threshold for establishing user-facing quality targets (SLA/SLO). |
| p99 | 99% of requests were faster. Only 1% (1 in 100) was slower. | Detecting rare but severe system failures, cold starts, and extreme queues. |
Suppose we measure a hypothetical series of 100 requests. If the p50 is at 800 ms, the p95 at 1,200 ms, and the p99 at 14,500 ms, this means that most users have a very fast experience, but 1 in 100 users has to wait nearly 15 seconds. The average for this series would be around 1,100 ms, leaving the 14.5-second p99 spike completely hidden in high-level metrics.
When measuring LLM latency, distinguishing between two phases of the generation process is essential: input processing (prefill) and output generation (decode). These two phases have completely different performance profiles.
TTFT is the time that elapses between the client sending the HTTP request and receiving the very first token of the LLM response. TTFT includes network latency, provider queue time, and the prefill phase in which the model processes the entire input prompt all at once.
A high TTFT almost always points to long input prompts, provider queue congestion, or network latency. For the interactive user experience (such as a chat interface), TTFT is the most critical metric: it determines how quickly the user sees that the system 'starts typing'.
The total response time is the duration from the request until the receipt of the very last token (or the closure of the stream). However, the total time depends heavily on how many tokens the model generated. A response of 500 tokens naturally takes longer than a response of 20 tokens.
Therefore, we split the measurement into two distinct components:
(Totale tijd - TTFT) / (Aantal gegenereerde tokens - 1). Alternatively, this is often expressed as generation speed in tokens per second (TPS).When comparing two LLM providers or two model versions, you cannot simply compare the absolute total response times side by side. If Model A returns a 50-token response in 2 seconds, and Model B takes 4 seconds but generates 300 tokens, then Model B is significantly faster during the generation phase (75 tokens/sec vs 25 tokens/sec), even though Model B's absolute response time is twice as high.
Without normalization, you end up with a comparison that depends on the accidental length of the generated response at that specific moment. Consult the comprehensive step-by-step guide for building your own benchmark for more guidelines on setting up standardized test suites.
To compare latency accurately at the percentile level, apply the following normalization steps:
One of the most common mathematical errors in monitoring dashboards is averaging percentiles. Suppose your application runs across five different servers. Each server calculates its own p95 latency every hour. Many developers tend to take the average of those five p95 values to determine the 'overall p95' for the hour. This is statistically invalid.
Percentiles are non-linear and non-additive. The p95 of a combined dataset can be significantly higher or lower than the average of the individual p95 values, depending on the volume and distribution of measurements across servers.
Suppose Server A handles 1,000 requests with a p95 of 1,000 ms. Server B handles only 10 requests, but due to an incident, its p95 was 20,000 ms. If you take the simple average of the two p95s, you get (1.000 + 20.000) / 2 = 10.500 ms. In reality, 950 of the 1,010 total requests fell well below 1,000 ms, putting the actual combined p95 close to 1,000 ms. Averaging percentiles provides an entirely misleading picture here.
In modern AI architectures, an LLM call rarely stands alone. Consider a Retrieval-Augmented Generation (RAG) pipeline where an embedding model is called first, followed by a query to a vector database, then a reranker, and finally the main model generating a response. In a multi-agent system, dozens of LLM calls may be executed sequentially or in parallel.
When multiple steps execute sequentially, percentiles compound in a dangerous way. The overall p99 of a pipeline is drastically worse than the p99 of the individual components.
If a pipeline consists of $n$ independent steps, and each step has a p99 latency of 1 second (meaning 1% of calls are very slow), then the probability that a request experiences no p99 delay is equal to $0{,}99^n$.
| Number of sequential steps ($n$) | Probability that everything runs fast ($0{,}99^n$) | Probability that at least 1 step hits the p99 tail |
|---|---|---|
| 1 step | 99,0% | 1,0% |
| 5 steps | 95,1% | 4,9% |
| 10 steps | 90,4% | 9,6% |
| 50 steps (complex agentic process) | 60,5% | 39,5% |
In a 50-step process, nearly 40% of end users will experience the p99 latency of at least one subsystem! This phenomenon is known as tail latency amplification. For this reason, the p95 and p99 requirements for individual building blocks in a RAG pipeline must be far stricter than the final SLO promised to the end user.
A common mistake in benchmarking is reporting a p99 based on an insufficient sample size. If you run a benchmark with 50 requests and calculate "the p99" from that, the result is pure statistical noise.
After all, the 99th percentile represents an event that occurs in only 1 out of 100 cases. To determine the 99th percentile with acceptable confidence, you need an absolute minimum number of observations.
Running a quick test of 30 prompts to try out a new model? Look exclusively at p50 and possibly p90. Do not draw conclusions about a provider's p99 based on a few dozen test requests.
Where do you measure latency? In practice, there is a significant difference between what the API provider reports in its status dashboard (server-side latency) and what the end user of your application actually experiences (client-side latency).
The provider measures the time from the moment the request hits their load balancers until the moment the last token leaves their network port. What is missing here:
Client-side latency encompasses the full round-trip time (RTT), including any delays in processing the HTTP response by your own SDK or application code. Furthermore, a local garbage collection pause in Node.js or Python can artificially inflate measured latency.
The way your application handles errors and network streams has a direct impact on measured percentiles.
When an LLM request fails with an HTTP 503 (Capacity Exceeded) or an HTTP 429 (Rate Limit), a robust client will retry the request with a short delay (exponential backoff). More details on configuring this can be found on the page about retry and backoff strategies.
If you only record latency once the request finally succeeds, the waiting time of failed attempts gets included in the total duration. A request that fails twice and only succeeds on the third attempt shoots straight into the p99 tail. Therefore, record both the latency of the attempt (per HTTP call) and the latency of the transaction (the total time the end user waits, including retries).
If you configure a hard client-side timeout at, for example, 10 seconds, all requests taking longer than 10 seconds are aborted and marked as an error. For handling this, see the guide on timeouts and cancellation.
If you subsequently remove these aborted requests from your latency metrics, it introduces truncation bias: your p99 suddenly appears artificially low (under 10 seconds), simply because you cut off the truly slow responses. Include aborted requests with the maximum configured timeout value in your percentile calculations, or explicitly report the error rate alongside your percentiles.
When using streaming HTTP responses, many client SDKs or proxies only forward tokens to the application once a buffer of a specific size (e.g., 4 KB) is full. This negatively skews the TTFT measurement. For latency measurements, make sure network buffers on the client are disabled (unbuffered stream) to measure the actual physical TTFT.
Moving from averages to percentiles enables you to agree on realistic Service Level Objectives (SLOs) with the team or the business. An agreement like *"The LLM service has an average response time of 1.5 seconds"* is worthless because it provides no guarantees for individual requests.
A professional percentile-based SLO definition looks like this:
Example of a robust LLM SLO:
"In 95% of cases over a rolling 30-day window, the Time To First Token (TTFT) must be under 800 ms, and the Time Per Output Token (TPOT) must be below 40 ms/token. Requests taking longer than 15 seconds are considered failed."
Setting a threshold at p95 automatically means accepting that 5% of requests may exceed the target. This is your error budget. Only when more than 5% of requests are slower than the p95 threshold (or when the p99 exceeds a critical upper bound of, say, 20 seconds) does the error budget become exhausted.
At that point, the team has a clear trigger to implement measures such as:
To conclude, an overview of the most common pitfalls in practice, and what the right approach is:
| Flawed approach | Why it fails | The correct method |
|---|---|---|
| Steering by average latency. | Hides the slow experience of tail users and reacts heavily to outliers. | Steer by p50 for the median experience and p95/p99 for the tail. |
| Averaging p95 values across multiple servers. | Statistically invalid; skews the true percentile. | Store raw data or use t-digest / HDR Histograms for aggregation. |
| Comparing total time without adjusting for token length. | Models generating longer responses falsely appear 'slower'. | Split into TTFT (ms) and TPOT (ms/token or TPS). |
| Reporting p99 based on 30 test requests. | A sample size that is too small leads to arbitrary noise. | Only evaluate p99 with at least 1,000 representative requests. |
| Discarding timeouts from the latency dataset. | Leads to truncation bias, making the p99 appear artificially low. | Count canceled/timed-out requests against the maximum duration. |
By applying percentiles consistently and correctly across your evaluations, benchmarks, and production monitoring, you transform latency from a vague impression into a hard, actionable quality metric for your AI infrastructure.
LLM-as-a-Judge: Models Judging ModelsDiscover how 'LLM-as-a-judge' works, what the bias risks are, and when automatic evaluation of AI models…
Setting up human evaluation: annotation guidelines that workDiscover how to set up effective annotation guidelines for human LLM evaluation. Learn about…
Evaluating Multimodal AI: Measuring ImageDiscover how to evaluate multimodal AI on image and audio input. Learn why text benchmarks fail and how…