# Measuring Latency with Percentiles Instead of Averages โ€” LLM Benchmark [Skip to content](#lm-inhoud)Network/[NL](/en/latency-percentielen-meten)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl โ€” main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Flatency-percentielen-meten&text=Measuring%20Latency%20with%20Percentiles%20Instead%20of%20Averages%20%E2%80%94%20LLM%20Benchmark)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Flatency-percentielen-meten)[](https://www.reddit.com/submit?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Flatency-percentielen-meten&title=Measuring%20Latency%20with%20Percentiles%20Instead%20of%20Averages%20%E2%80%94%20LLM%20Benchmark)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Flatency-percentielen-meten&text=Measuring%20Latency%20with%20Percentiles%20Instead%20of%20Averages%20%E2%80%94%20LLM%20Benchmark)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Flatency-percentielen-meten)[](https://www.reddit.com/submit?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Flatency-percentielen-meten&title=Measuring%20Latency%20with%20Percentiles%20Instead%20of%20Averages%20%E2%80%94%20LLM%20Benchmark)[](#) # Measuring latency with percentiles instead of averages By Ivo Donker โ€” compiled with AI support (Claude & Gemini) ยท Last updated: August 7, 2026 When evaluating Large Language Model (LLM) API performance, latency is one of the most critical operational metrics. Yet many engineering teams and technical decision-makers fall into the trap of expressing latency as a single average. While average response time can already provide a skewed picture in traditional web applications, the arithmetic mean is downright misleading for LLM infrastructure. LLM interactions do not follow a classic normal distribution (the familiar bell curve), but rather a heavily skewed distribution with an extremely long tail (a so-called heavy-tailed distribution). Making decisions based on averages ignores the frustrating experience of the 5 to 10 percent of users who encounter unacceptably slow requests. In this guide, we cover why the latency distribution in LLMs is so skewed, how to interpret percentiles (p50, p90, p95, p99), how to properly normalize measurements for output length, and which pitfalls exist when aggregating data and establishing a Service Level Objective (SLO). ## Why averages fail for LLM latency The arithmetic mean is calculated by dividing the sum of all measured response times by the total number of measurements. This metric is extremely sensitive to outliers. When nineteen requests are completed within 500 milliseconds, but the twentieth takes 30 seconds due to a cold start mechanism or retry, the average across all twenty requests instantly jumps to nearly 2 seconds. However, that 2-second average fails to tell you two things: - It does not reflect what the typical user experiences (who only waits 500 milliseconds). - It hides the severity of the problem for the unlucky user (who waits 30 seconds for a timeout or response). For a broader overview of setting up performance benchmarks, you can consult the guide on [measuring speed in LLMs](/en/snelheid-meten) , but the root cause of this specific latency behavior lies in the architecture of AI cloud providers. ### The causes of the long tail in LLM APIs The response time of an LLM endpoint is not determined solely by the compute required for a prompt, but by a chain of dynamic factors at the provider level: - Queuing and KV cache eviction: When a provider experiences a sudden surge in requests, incoming calls end up in a queue (queuing delay). If the GPU's Key-Value (KV) cache becomes full, earlier contexts must be evicted and later reloaded from host memory, causing a massive latency spike. - Dynamic capacity allocation and preemption: Providers continuously scale instances. When your request lands on a GPU that is just initializing or where another request takes precedence (preemption), latency increases. - Cold starts of specific adapters: Invoking less frequently used fine-tuned models or specific LoRA adapters requires the weights file to be loaded dynamically into the inference server's VRAM. - Network jitter and TCP retransmissions: Because LLM responses are often sent over streaming HTTP connections (Server-Sent Events), a single dropped network packet causes delays across the entire stream. ## What p50, p95, and p99 actually mean To properly understand the distribution of response times, we use percentiles. A percentile is a threshold indicating what percentage of measurements falls at or below that value when all measurements are sorted in ascending order. Percentile | Practical meaning | Operational focus | p50 (Median) | 50% of requests were faster than this value, and 50% were slower. This represents the 'typical' experience. | Insight into the overall efficiency of the infrastructure under normal conditions. | p90 | 90% of requests were faster than this value; 10% experienced higher latency. | First indication of building traffic or minor capacity constraints. | p95 | 95% of requests fell within this time. 1 in 20 requests was slower. | The standard threshold for establishing user-facing quality targets (SLA/SLO). | p99 | 99% of requests were faster. Only 1% (1 in 100) was slower. | Detecting rare but severe system failures, cold starts, and extreme queues. | Suppose we measure a hypothetical series of 100 requests. If the p50 is at 800 ms, the p95 at 1,200 ms, and the p99 at 14,500 ms, this means that most users have a very fast experience, but 1 in 100 users has to wait nearly 15 seconds. The average for this series would be around 1,100 ms, leaving the 14.5-second p99 spike completely hidden in high-level metrics. ## The difference between TTFT and total response time When measuring LLM latency, distinguishing between two phases of the generation process is essential: input processing (prefill) and output generation (decode). These two phases have completely different performance profiles. ### 1. Time To First Token (TTFT) TTFT is the time that elapses between the client sending the HTTP request and receiving the very first token of the LLM response. TTFT includes network latency, provider queue time, and the prefill phase in which the model processes the entire input prompt all at once. A high TTFT almost always points to long input prompts, provider queue congestion, or network latency. For the interactive user experience (such as a chat interface), TTFT is the most critical metric: it determines how quickly the user sees that the system 'starts typing'. ### 2. Total response time and Time Per Output Token (TPOT) The total response time is the duration from the request until the receipt of the very last token (or the closure of the stream). However, the total time depends heavily on how many tokens the model generated. A response of 500 tokens naturally takes longer than a response of 20 tokens. Therefore, we split the measurement into two distinct components: - TTFT (Time To First Token): Measured in milliseconds. - TPOT (Time Per Output Token) / Inter-token latency: Measured in milliseconds per token, calculated as (Totale tijd - TTFT) / (Aantal gegenereerde tokens - 1). Alternatively, this is often expressed as generation speed in tokens per second (TPS). Please note: Always analyze TTFT and TPOT as separate percentile distributions. A model can have an excellent p50 TTFT (200 ms) but a poor p95 TPOT (80 ms per token), causing long responses to still feel sluggish halfway through. ## Why you must normalize output length When comparing two LLM providers or two model versions, you cannot simply compare the absolute total response times side by side. If Model A returns a 50-token response in 2 seconds, and Model B takes 4 seconds but generates 300 tokens, then Model B is significantly faster during the generation phase (75 tokens/sec vs 25 tokens/sec), even though Model B's absolute response time is twice as high. Without normalization, you end up with a comparison that depends on the accidental length of the generated response at that specific moment. Consult the comprehensive [step-by-step guide for building your own benchmark](/en/eigen-benchmark-opzetten-stappenplan) for more guidelines on setting up standardized test suites. To compare latency accurately at the percentile level, apply the following normalization steps: - Group benchmarks based on a fixed prompt length (e.g., prompts of ~500 tokens). - Evaluate the p50, p95, and p99 of the TTFT to compare prompt processing throughput. - Evaluate the p50, p95, and p99 of the TPOT (or TPS) to compare generation speed. ## The aggregation problem: you cannot average percentiles One of the most common mathematical errors in monitoring dashboards is averaging percentiles. Suppose your application runs across five different servers. Each server calculates its own p95 latency every hour. Many developers tend to take the average of those five p95 values to determine the 'overall p95' for the hour. This is statistically invalid. Percentiles are non-linear and non-additive. The p95 of a combined dataset can be significantly higher or lower than the average of the individual p95 values, depending on the volume and distribution of measurements across servers. ### Fictional numerical example of the aggregation error Suppose Server A handles 1,000 requests with a p95 of 1,000 ms. Server B handles only 10 requests, but due to an incident, its p95 was 20,000 ms. If you take the simple average of the two p95s, you get (1.000 + 20.000) / 2 = 10.500 ms. In reality, 950 of the 1,010 total requests fell well below 1,000 ms, putting the actual combined p95 close to 1,000 ms. Averaging percentiles provides an entirely misleading picture here. ### What should you do instead? - Store raw measurements: Store the exact duration of each request in a time-series database (such as ClickHouse or Datadog) and compute the percentile across the full dataset for the desired time window. - Use data structures for estimation: If storing every individual data point is cost-prohibitive, use data structures such as t-digest or HDR Histograms. These allow you to merge percentiles from different nodes and time windows in a mathematically sound way. For more background on this, see the article on [statistics for evaluations](/en/statistiek-voor-evaluaties). ## Cascading effects: how the tail builds up in RAG and agents In modern AI architectures, an LLM call rarely stands alone. Consider a Retrieval-Augmented Generation (RAG) pipeline where an embedding model is called first, followed by a query to a vector database, then a reranker, and finally the main model generating a response. In a multi-agent system, dozens of LLM calls may be executed sequentially or in parallel. When multiple steps execute sequentially, percentiles compound in a dangerous way. The overall p99 of a pipeline is drastically worse than the p99 of the individual components. If a pipeline consists of $n$ independent steps, and each step has a p99 latency of 1 second (meaning 1% of calls are very slow), then the probability that a request experiences no p99 delay is equal to $0{,}99^n$. Number of sequential steps ($n$) | Probability that everything runs fast ($0{,}99^n$) | Probability that at least 1 step hits the p99 tail | 1 step | 99,0% | 1,0% | 5 steps | 95,1% | 4,9% | 10 steps | 90,4% | 9,6% | 50 steps (complex agentic process) | 60,5% | 39,5% | In a 50-step process, nearly 40% of end users will experience the p99 latency of at least one subsystem! This phenomenon is known as tail latency amplification. For this reason, the p95 and p99 requirements for individual building blocks in a RAG pipeline must be far stricter than the final SLO promised to the end user. ## Sample size: how many measurements do you need? A common mistake in benchmarking is reporting a p99 based on an insufficient sample size. If you run a benchmark with 50 requests and calculate "the p99" from that, the result is pure statistical noise. After all, the 99th percentile represents an event that occurs in only 1 out of 100 cases. To determine the 99th percentile with acceptable confidence, you need an absolute minimum number of observations. - For p50 (Median): A minimum of 20 to 30 representative measurements. - For p95: A minimum of 100 to 200 measurements. With 100 measurements, the p95 is simply the sixth-slowest value. A single incidental outlier can heavily distort the value. - For p99: A minimum of 1,000 measurements. Only at 1,000 requests do you have 10 data points in the tail, which is necessary to distinguish structural tail latency from an accidental outlier. Running a quick test of 30 prompts to try out a new model? Look exclusively at p50 and possibly p90. Do not draw conclusions about a provider's p99 based on a few dozen test requests. ## Client-Side vs. Server-Side Measurements Where do you measure latency? In practice, there is a significant difference between what the API provider reports in its status dashboard (server-side latency) and what the end user of your application actually experiences (client-side latency). ### Server-Side Latency (Provider Perspective) The provider measures the time from the moment the request hits their load balancers until the moment the last token leaves their network port. What is missing here: - DNS resolution by the client. - TLS handshake and TCP connection setup. - Network transit time between your server (e.g., in AWS eu-west-1) and the provider's GPU cluster (e.g., in a data center elsewhere). ### Client-Side Latency (Your Perspective) Client-side latency encompasses the full round-trip time (RTT), including any delays in processing the HTTP response by your own SDK or application code. Furthermore, a local garbage collection pause in Node.js or Python can artificially inflate measured latency. Benchmarking Recommendation: For decision-making, always use client-side measurements taken from the exact location and network environment where your application will run in production. For detailed instructions on setting up your observability, consult the guide on [observability and logging for LLM APIs](https://api.llmnet.nl/en/observability-en-logging) . ## The Impact of Retries, Timeouts, and Streaming The way your application handles errors and network streams has a direct impact on measured percentiles. ### 1. Retries and Backoff When an LLM request fails with an HTTP 503 (Capacity Exceeded) or an HTTP 429 (Rate Limit), a robust client will retry the request with a short delay (exponential backoff). More details on configuring this can be found on the page about [retry and backoff strategies](https://api.llmnet.nl/en/retries-en-backoff). If you only record latency once the request finally succeeds, the waiting time of failed attempts gets included in the total duration. A request that fails twice and only succeeds on the third attempt shoots straight into the p99 tail. Therefore, record both the latency of the attempt (per HTTP call) and the latency of the transaction (the total time the end user waits, including retries). ### 2. Timeouts and leniency bias If you configure a hard client-side timeout at, for example, 10 seconds, all requests taking longer than 10 seconds are aborted and marked as an error. For handling this, see the guide on [timeouts and cancellation](https://api.llmnet.nl/en/timeouts-en-cancellation). If you subsequently remove these aborted requests from your latency metrics, it introduces truncation bias: your p99 suddenly appears artificially low (under 10 seconds), simply because you cut off the truly slow responses. Include aborted requests with the maximum configured timeout value in your percentile calculations, or explicitly report the error rate alongside your percentiles. ### 3. Streaming buffering When using streaming HTTP responses, many client SDKs or proxies only forward tokens to the application once a buffer of a specific size (e.g., 4 KB) is full. This negatively skews the TTFT measurement. For latency measurements, make sure network buffers on the client are disabled (unbuffered stream) to measure the actual physical TTFT. ## Translating to SLOs, SLAs, and Error Budgets Moving from averages to percentiles enables you to agree on realistic Service Level Objectives (SLOs) with the team or the business. An agreement like *"The LLM service has an average response time of 1.5 seconds"* is worthless because it provides no guarantees for individual requests. A professional percentile-based SLO definition looks like this: Example of a robust LLM SLO: "In 95% of cases over a rolling 30-day window, the Time To First Token (TTFT) must be under 800 ms, and the Time Per Output Token (TPOT) must be below 40 ms/token. Requests taking longer than 15 seconds are considered failed." ### Linking an Error Budget Setting a threshold at p95 automatically means accepting that 5% of requests may exceed the target. This is your error budget. Only when more than 5% of requests are slower than the p95 threshold (or when the p99 exceeds a critical upper bound of, say, 20 seconds) does the error budget become exhausted. At that point, the team has a clear trigger to implement measures such as: - Switching to a smaller or faster model for specific tasks. - Enabling more aggressive prompt-level caching. - Using a fallback provider during elevated latency. ## Common mistakes in LLM latency measurements To conclude, an overview of the most common pitfalls in practice, and what the right approach is: Flawed approach | Why it fails | The correct method | Steering by average latency. | Hides the slow experience of tail users and reacts heavily to outliers. | Steer by p50 for the median experience and p95/p99 for the tail. | Averaging p95 values across multiple servers. | Statistically invalid; skews the true percentile. | Store raw data or use t-digest / HDR Histograms for aggregation. | Comparing total time without adjusting for token length. | Models generating longer responses falsely appear 'slower'. | Split into TTFT (ms) and TPOT (ms/token or TPS). | Reporting p99 based on 30 test requests. | A sample size that is too small leads to arbitrary noise. | Only evaluate p99 with at least 1,000 representative requests. | Discarding timeouts from the latency dataset. | Leads to truncation bias, making the p99 appear artificially low. | Count canceled/timed-out requests against the maximum duration. | By applying percentiles consistently and correctly across your evaluations, benchmarks, and production monitoring, you transform latency from a vague impression into a hard, actionable quality metric for your AI infrastructure.