Measuring time to first token under varying server load

The responsiveness of a language model application stands or falls with the perceived initial wait time. The core metric for this is Time to First Token (TTFT): the exact duration between the moment a client sends the HTTP request and the moment the very first token arrives via the Server-Sent Events (SSE) stream. Where total end-to-end latency depends heavily on the number of output tokens generated, TTFT purely reflects the combination of network latency, server-side queuing delay, and the computational processing of the input prompt. With this measurement, you make concrete infrastructure decisions about batching algorithms, capacity planning, and fallback strategies to alternative model instances when queues fill up.

In this article, we explicitly distinguish TTFT measurement from general speed statistics such as average throughput per second. Where the general fundamentals are described in the overview of latency, throughput, and tokens per second, this guide focuses specifically on the dynamics of streaming prefill delays under concurrent system pressure. A single request on an idle GPU, after all, gives a distorted picture of reality. Only when multiple parallel users simultaneously send context windows of varying size does it become visible how queue management and memory allocation affect the initial wait time.

The anatomy of time to first token

To understand why TTFT explodes under load, we need to break down the lifecycle of an LLM call into separate hardware and software steps. A language model fundamentally processes data in two phases: the prefill phase and the decoding phase. To understand which matrix computations take place on the GPU during these phases, see the article where the technical workings of AI inference under the hood are explained step by step. During the prefill phase, the model reads in the entire input prompt in a single parallel computation step. Here, the GPU calculates all key and value vectors (KV cache) for the context. This process is heavily compute-bound.

The decoding phase, by contrast, generates one token at a time, sequentially. This is primarily memory-bandwidth-bound. TTFT encompasses the full network round trip, the time the request spends in the serving engine's application queue, the prefill computation of the prompt, and the generation of the very first output token. As soon as the server is processing hundreds of streams concurrently, competition arises for GPU memory and compute units, making queue time the dominant factor within TTFT.

Why averages fail: measuring in percentiles

Reporting an average TTFT over a test period structurally masks the occasional spikes that end users run into. When an inference engine runs under continuous load, sudden spikes in concurrency or periodic memory fragmentation (deallocation of KV caches) cause so-called "tail latency spikes." An average TTFT of 400 milliseconds can coexist with a 99th percentile of 8 seconds, meaning that one in a hundred interactions feels like a freeze to a human user.

For reliable benchmarking, we therefore look exclusively at percentile distributions: p50 (median), p90, p95, and p99. Anyone who wants to dig deeper into the statistical justification for these distributions can consult the methodology for measuring latency percentiles instead of averages . When benchmarking TTFT under varying server load, we compare how the curve between p50 and p99 fans out as the number of virtual users increases. A healthy architecture shows a p99 that grows in parallel with p50; an unstable queue shows a p99 that runs away exponentially while p50 appears to stay flat.

Factors affecting TTFT during peak load

As server load increases, several internal mechanisms of the serving engine (such as vLLM, TensorRT-LLM, or TGI) kick in simultaneously. To interpret test results accurately, we need to isolate the four main causes of TTFT degradation:

• Continuous batching and chunked prefill: Modern engines mix prefill and decoding tasks within the same iteration step. Because prefill operations claim all compute cores, engines split large prompts into smaller chunks (chunked prefill). This prevents ongoing decoding streams from stuttering, but directly increases the TTFT of the incoming prompt.
• KV-cache memory pressure and preemption: Once video memory fills up with active conversation histories, the scheduler has to make choices. New requests wait in a queue until earlier streams finish, or ongoing requests are temporarily paused (swapping/preemption), leading to extreme TTFT outliers.
• Prompt caching hit ratio: When multiple requests share the same system prompt or document context, the engine can reuse earlier KV computations. A cache hit can lower TTFT by as much as 80%, while a cache miss under heavy load has to go through the full prefill.
• Network and TLS handshakes: When testing over public internet connections, TCP slow-start and TLS renegotiation can add tens of milliseconds that have nothing to do with model performance.

Setting up a reproducible measurement design

A scientifically sound benchmark requires strict control over variables. If input length fluctuates randomly between 50 and 4,000 tokens, you're not measuring server degradation but random variance in prompt length. For a reproducible TTFT load test, we therefore follow a standardized protocol:

1. Fixed prompt lengths: Create synthetic payloads with exactly defined token counts (for example, 256, 1,024, and 4,096 tokens). Use a fixed seed value for random text generation to ensure contextual consistency.
2. Controlled concurrency steps: Don't test with a random "burst," but ramp up the load step by step. Start with 1 virtual user (the unloaded baseline measurement), and increase in fixed steps (e.g., 5, 10, 20, 50 concurrent workers) with a minimum of 3 minutes per step to reach thermal throttling and steady-state memory occupancy.
3. Eliminating client-side bottlenecking: Make sure the test script runs asynchronously on a dedicated machine with sufficient network bandwidth and CPU power, so the client itself doesn't introduce delay when parsing incoming streaming chunks.
4. Excluding client caching: Deliberately enable HTTP keep-alive to mimic real production situations, but vary unique identifiers if desired to bypass any model-level prompt caches when you want to test raw compute capacity.

Implementation: benchmarking TTFT with Python and AsyncIO

The script below illustrates how, using Python, httpx and asyncio you run a controlled load test that specifically records the streaming time to the first chunk. Note the exact timing: we start the clock right before sending the HTTP POST payload and stop the clock as soon as the first non-empty line arrives from the streaming response.

import asyncio
import time
import numpy as np
import httpx

API_URL = "http://localhost:8000/v1/chat/completions"
HEADERS = {"Authorization": "Bearer test-key", "Content-Type": "application/json"}

# Vaste prompt om ruis in prefill-berekeningen te voorkomen
PROMPT_PAYLOAD = {
  "model": "meta-llama/Llama-3-8B-Instruct",
  "messages": [
    {"role": "system", "content": "Je bent een behulpzame assistent."},
    {"role": "user", "content": "Schrijf een technisch essay van 500 woorden over netwerkprotocollen."}
  ],
  "stream": True,
  "max_tokens": 100,
  "temperature": 0.0
}

async def measure_single_ttft(client: httpx.AsyncClient) -> float:
  start_time = time.perf_counter()
  first_token_time = None
  
  try:
    async with client.stream("POST", API_URL, json=PROMPT_PAYLOAD, headers=HEADERS, timeout=60.0) as response:
      if response.status_code != 200:
        return None
      async for line in response.aiter_lines():
        if line.startswith("data: ") and line.strip() != "data: [DONE]":
          first_token_time = time.perf_counter()
          break
  except Exception:
    return None

  if first_token_time:
    return (first_token_time - start_time) * 1000.0  # Milliseconden
  return None

async def run_concurrency_tier(concurrency: int, total_requests: int):
  limits = httpx.Limits(max_keepalive_connections=concurrency, max_connections=concurrency * 2)
  async with httpx.AsyncClient(limits=limits) as client:
    semaphore = asyncio.Semaphore(concurrency)
    
    async def worker():
      async with semaphore:
        return await measure_single_ttft(client)

    tasks = [worker() for _ in range(total_requests)]
    results = await asyncio.gather(*tasks)
    
  valid_ttfts = [r for r in results if r is not None]
  if not valid_ttfts:
    print(f"Concurrency {concurrency}: Alle requests gefaald.")
    return

  print(f"--- Concurrency Niveau: {concurrency} workers ({len(valid_ttfts)}/{total_requests} geslaagd) ---")
  print(f"p50 TTFT: {np.percentile(valid_ttfts, 50):.1f} ms")
  print(f"p90 TTFT: {np.percentile(valid_ttfts, 90):.1f} ms")
  print(f"p95 TTFT: {np.percentile(valid_ttfts, 95):.1f} ms")
  print(f"p99 TTFT: {np.percentile(valid_ttfts, 99):.1f} ms")

async def main():
  for workers in [1, 5, 10, 25]:
    await run_concurrency_tier(concurrency=workers, total_requests=workers * 10)
    await asyncio.sleep(2)

if __name__ == "__main__":
  asyncio.run(main())

The interplay with payload integrity

When an LLM server buckles under heavy concurrency, it's often not just TTFT that degrades; the stability of the inference engine itself also comes under pressure. Under extreme queue contention and abruptly dropped TCP connections, we regularly see streaming chunks become incomplete or the server's internal parser produce corrupted output. Anyone enforcing strict JSON schemas in production can consult the evaluation method for JSON formatting validity under high concurrency to verify whether enforcing syntactic correctness adds extra delay to TTFT.

The prefill phase for JSON geometries or extensive tool-calling definitions requires extra grammar preprocessing (as with outlines or constrained sampling). Under varying server load, this constrained decoding creates a double burden: TTFT rises due to the initial schema compilation, while decoding throughput stagnates because invalid token logits must be masked at every step.

Dutch-language pitfalls in TTFT benchmarking

When benchmarking Dutch-language prompts against multilingual models, we run into a specific linguistic anomaly: tokenization inefficiency. Because most tokenizers are trained on predominantly English-language corpora, Dutch words are often split into considerably more subtokens per word than their English equivalents. Compound words such as "aansprakelijkheidsverzekeringsmaatschappij" or common verb forms result in a suboptimally fragmented token sequence.

This has direct consequences for TTFT under server load. A seemingly identical prompt of 300 Dutch words can result in 550 input tokens, while the English version counts only 380 tokens. During the compute-bound prefill phase, the GPU therefore has to perform substantially more matrix multiplications for exactly the same semantic payload. When dozens of concurrent users send Dutch-language prompts at the same time, the server's prefill capacity saturates up to 40% faster than with English-language prompts. Always include representative Dutch business texts in your benchmark set, therefore, rather than directly translated short English sentences.

Cost, turnaround time, and capacity limits

Running large-scale load tests on a regular basis carries significant operational costs, both in API credit and required test time. Generating 10,000 test requests with context windows of 2,048 tokens pushes millions of tokens through the pipeline. To avoid budget surprises, it's advisable to calculate in advance how many requests are strictly necessary to compute statistically reliable percentiles.

When commercial API gateways are tested, external rate limiting also often kicks in before the hardware TTFT limit of the underlying cluster is reached. To understand how intermediate proxies throttle requests using algorithms such as leaky bucket or token bucket, we look at how one applies a token bucket in an LLM gateway to smooth out traffic flows. A good load test measures purely the raw response time of the inference nodes by explicitly monitoring and logging rate limits from intermediate layers via HTTP 429 error codes.

Conclusion and checklist for reliable TTFT measurements

Measuring TTFT under varying system load ruthlessly exposes how an AI architecture behaves when it really matters. By resolutely rejecting averages and steering by p95 and p99 percentiles, you discover early where prefill bottlenecks, memory fragmentation, and queue buildup occur. Apply the following fixed steps in every new benchmark cycle:

• Isolate TTFT from total turnaround time by explicitly listening for the first streaming packet via Server-Sent Events.
• Use identical, fixed prompt sizes to rule out variance in computational prefill duration.
• Ramp up the load step by step through clear concurrency tiers, and measure for at least a few minutes per tier.
• Always report p50, p90, p95, and p99 alongside the percentage of failed or timed-out connections.
• Account for Dutch tokenization skew when sizing input payloads.

With this measurement method, you turn superficial speed claims into hard, reproducible data, giving you a solid basis for decisions about hardware deployment, serving frameworks, and scalability limits.