Latency, throughput and tokens/second

Introduction

Measuring the speed of Large Language Models (LLMs) differs fundamentally from traditional web applications. While a web server returns a complete HTTP response in milliseconds, LLMs generate text token by token in an iterative process. This requires specialized metrics and a careful measurement setup to compare apples to apples.

The Core Metrics

To accurately map the performance of an LLM endpoint, three fundamental pillars are examined:

What affects speed?

Several factors at the hardware, model, and network level directly influence the measured performance:

• Hardware and Quantization: VRAM bandwidth (e.g., HBM3e vs. GDDR6) is often the primary bottleneck. Quantization (such as INT4 or FP8) reduces memory footprint and increases tokens/second, provided the hardware is optimized for it.
• Prompt Length (Context Window): The prefill phase (processing the input prompt) scales differently than the generation phase. Longer inputs significantly increase the TTFT.
• Concurrency and Batching: Dynamic batching on the inference engine (such as vLLM or TGI) maximizes GPU utilization under heavy load, which affects individual latency.
• Network Latency: Physical distance to the data center and the implementation of server-sent events (SSE) or WebSockets affect the perceived response time.

Example Measurement Setup

For a fair and reproducible benchmark, it is necessary to control external variables. Below is a conceptual script to perform standardized measurements via an API.

import time
import requests

def benchmark_endpoint(api_url, payload, api_key, iterations=5):
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
results = []

for i in range(iterations):
start_time = time.perf_counter()
first_token_time = None
total_tokens = 0

response = requests.post(api_url, json=payload, headers=headers, stream=True)

for line in response.iter_lines():
if line:
if first_token_time is None:
first_token_time = time.perf_counter()
# Count tokens based on chunk structure (provider-specific)
total_tokens += 1

end_time = time.perf_counter()

ttft = first_token_time - start_time
total_duration = end_time - start_time
generation_time = total_duration - ttft
tokens_per_sec = total_tokens / generation_time if generation_time > 0 else 0

results.append({
"run": i + 1,
"ttft_sec": ttft,
"tokens_per_sec": tokens_per_sec
})

return results

Comparative Table

The table below structures the recorded measurements. Note: the values shown are placeholders and should be filled in based on actual, calibrated test runs.