# Prompt caching simulator: measuring cost and latency

[Skip to content](#lm-inhoud)Network/[NL](/en/simulator-voor-prompt-caching-kostenbesparing-en-latency)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Fsimulator-voor-prompt-caching-kostenbesparing-en-latency&text=Prompt%20caching%20simulator%3A%20measuring%20cost%20and%20latency)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Fsimulator-voor-prompt-caching-kostenbesparing-en-latency)[](https://www.reddit.com/submit?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Fsimulator-voor-prompt-caching-kostenbesparing-en-latency&title=Prompt%20caching%20simulator%3A%20measuring%20cost%20and%20latency)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Fsimulator-voor-prompt-caching-kostenbesparing-en-latency&text=Prompt%20caching%20simulator%3A%20measuring%20cost%20and%20latency)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Fsimulator-voor-prompt-caching-kostenbesparing-en-latency)[](https://www.reddit.com/submit?url=https%3A%2F%2Fbenchmark.llmnet.nl%2Fen%2Fsimulator-voor-prompt-caching-kostenbesparing-en-latency&title=Prompt%20caching%20simulator%3A%20measuring%20cost%20and%20latency)[](#)

 
# Simulator for prompt caching cost savings and latency

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 Prompt caching promises drastic reductions in API bills and lower response times for language models. Model vendors' marketing cites discounts of up to eighty or ninety percent on cached input tokens. In production, however, the actual saving depends heavily on variables such as the length of the static prefix, the volatility of user questions, reuse within the time to live (TTL) and the ratio between input and output tokens. This article offers a mathematical measurement setup and a simulation model for quantifying the real impact on cost and latency exactly, in advance.

 This article deliberately sets itself apart from general price comparisons such as those in the guide on [calculating cost per task across models](https://benchmark.llmnet.nl/en/kosten-per-taak). Where that article works from static token prices per transaction, this analysis focuses specifically on the dynamic transition between cold cache writes and warm cache reads under varying workloads.

 
## 1. The theoretical mechanism behind KV cache reuse

 When a transformer model processes a prompt, the hardware computes so-called key and value vectors (KV pairs) for every token across all attention layers. This computation scales quadratically or linearly with context length, depending on the attention architecture used. Without caching, the server has to push the complete prefix back through the transformer layers on every incoming request. This causes a considerable computational load before the first output token can even be generated.

 Prompt caching stores the computed KV vectors of a deterministic prefix in the fast memory (such as HBM or host RAM) of the server cluster. On a subsequent request with exactly the same prefix, the model does not have to recompute these tensors but reads them straight from storage. For a deeper dive into the underlying technique and API implementations we refer to the background article that dissects [context caching at API providers](https://hub.llmnet.nl/en/context-caching-uitgelegd) technically. Reusing these vectors reduces both the FLOPs needed on the GPU and the processing time of the input phase.

 
## 2. Mathematical modeling of cost and effective token prices

 To model the cost saving accurately, we have to split a request into three separate token categories: the static prefix that qualifies for caching ($T_p$), the dynamic part of the prompt ($T_d$), and the generated output ($T_o$). Total cost per call ($K$) depends on whether the prefix produces a cache hit or a cache miss.

 Let $P_w$ be the price per token for writing to the cache (often equal to or marginally higher than the standard input rate), $P_r$ the reduced price for reading from the cache, $P_i$ the standard rate for dynamic input tokens, and $P_o$ the rate for output tokens. At a cache hit rate $H$ (expressed as a fraction between 0 and 1), the formula for expected cost per individual request is:

 K_verwacht = (1 - H) * (T_p * P_w + T_d * P_i + T_o * P_o) + H * (T_p * P_r + T_d * P_i + T_o * P_o)

 Rewriting this equation shows that the saving occurs exclusively on the static prefix block:

 K_verwacht = T_p * (P_w - H * (P_w - P_r)) + T_d * P_i + T_o * P_o

 From this follows a fundamental limitation of prompt caching directly: when an application has relatively few input tokens but generates sizeable texts (where $T_o \gg T_p$), the percentage cost saving converges to zero, no matter how high the hit rate $H$ is.

 
 Note: The numbers in the examples and tables below serve purely to illustrate the mathematical calculation method and are not current measurements of specific commercial models.

 

 
## 3. Latency reduction: time to first token and throughput

 Besides cost, prompt caching has a direct effect on user experience through latency. Total model response time consists of two phases: the prefill phase (where input tokens are processed) and the decode phase (where tokens are generated one by one). The prefill phase determines time to first token (TTFT).

 On a cache miss, TTFT grows proportionally longer as the prefix grows. On a cache hit, the system skips the matrix multiplications for the prefix, so TTFT falls back almost to the time needed to process only the dynamic part ($T_d$), plus a small constant lookup latency for the KV cache ($L_{lookup}$):

 TTFT_hit = (T_d / Prefill_Doorvoer) + L_lookup
TTFT_miss = ((T_p + T_d) / Prefill_Doorvoer) + L_write

 To quantify these time intervals cleanly in production environments, it is advisable to consult the methodology for [measuring TTFT under varying server load](https://benchmark.llmnet.nl/en/time-to-first-token-meten-onder-varierende-serverbelasting). That lets you isolate network effects from the actual compute performance of the model cluster.

 
 
 
 
 Prefix length (tokens) | 
 Hit rate (H) | 
 Expected TTFT reduction | 
 Expected cost saving on input | 
 

 
 
 
 1.024 | 
 50% | 
 ~25% | 
 ~37% | 
 

 
 4.096 | 
 80% | 
 ~65% | 
 ~60% | 
 

 
 16.384 | 
 90% | 
 ~82% | 
 ~72% | 
 

 
 32.768 | 
 95% | 
 ~90% | 
 ~76% | 
 

 
 
 

 The actual speed gain is also closely tied to the overall throughput of the chosen inference engine. See also the overview of how to [measure latency and tokens per second systematically](https://benchmark.llmnet.nl/en/snelheid-meten) in order to separate prefill speed from generation speed.

 
## 4. Factors that affect the cache hit rate

 The assumption that a 90% hit rate is easily achievable often proves too optimistic in practice. A simulator has to account for several disruptive factors that lower the effective hit rate:

 1. The byte-identical prefix requirement: LLM caching mechanisms work on exact matching of token sequences from the start of the document. If a dynamic element (such as a variable user name, a random session ID or a current timestamp) is placed at the front of the system message, the entire subsequent cache is invalidated. All dynamic data must be placed strictly after the static prompt block.

 2. Eviction policies and TTL: Cached tensors are not kept indefinitely. Providers typically apply a time to live (TTL) of, say, 5 to 60 minutes. If the gaps between successive requests are longer than the TTL, a cache eviction occurs and the prefix has to be written again at the full (or increased) write rate.

 3. Routing and cluster distribution: At large cloud providers, API requests are distributed across hundreds of physical machines. If the load balancer applies no session affinity or consistent hashing at prefix level, an identical prompt can land on a machine where the KV cache does not yet exist, causing an unnecessary cache miss.

 In complex systems where multiple autonomous processes work together, cache efficiency can be raised considerably through smart design patterns. See the analysis of [caching architectures for multi-agent loops](https://radar.llmnet.nl/en/caching-architecturen-voor-complexe-multi-agent-loops), which discusses techniques for reusing static system definitions consistently across multiple agent steps.

 
## 5. A Python simulation model for workload analysis

 The script below demonstrates how to simulate a synthetic or historical workload in order to quantify the expected cost and latency reduction. The script models arrival times through a Poisson process, accounts for a TTL window and computes the difference between a cached and an uncached architecture.

 import math
import random

def simuleer_caching_werklast(
 aantal_verzoeken=1000,
 gem_interval_sec=120.0,
 ttl_sec=300.0,
 prefix_tokens=8000,
 dynamisch_invoer_tokens=400,
 uitvoer_tokens=250,
 prijs_schrijf_per_k=0.003,
 prijs_lees_per_k=0.00075,
 prijs_normaal_in_per_k=0.003,
 prijs_uit_per_k=0.015,
 prefill_tps=2500.0
):
 huidige_tijd = 0.0
 laatste_hit_tijd = -999999.0
 
 kosten_zonder_cache = 0.0
 kosten_met_cache = 0.0
 totale_ttft_zonder = 0.0
 totale_ttft_met = 0.0
 hits = 0
 misses = 0

 for _ in range(aantal_verzoeken):
 # Genereer aankomsttijd via exponentiële verdeling
 interval = -gem_interval_sec * math.log(1.0 - random.random())
 huidige_tijd += interval

 # Bereken kosten zonder caching
 k_in_basis = ((prefix_tokens + dynamisch_invoer_tokens) / 1000.0) * prijs_normaal_in_per_k
 k_uit_basis = (uitvoer_tokens / 1000.0) * prijs_uit_per_k
 kosten_zonder_cache += (k_in_basis + k_uit_basis)
 
 # TTFT zonder caching
 ttft_basis = (prefix_tokens + dynamisch_invoer_tokens) / prefill_tps
 totale_ttft_zonder += ttft_basis

 # Controleer cache status
 if (huidige_tijd - laatste_hit_tijd) <= ttl_sec:
 # Cache HIT
 hits += 1
 k_in_cache = ((prefix_tokens / 1000.0) * prijs_lees_per_k) + \
 ((dynamisch_invoer_tokens / 1000.0) * prijs_normaal_in_per_k)
 ttft_cache = (dynamisch_invoer_tokens / prefill_tps) + 0.015 # opzoekoverhead
 else:
 # Cache MISS (schrijven naar cache)
 misses += 1
 k_in_cache = ((prefix_tokens / 1000.0) * prijs_schrijf_per_k) + \
 ((dynamisch_invoer_tokens / 1000.0) * prijs_normaal_in_per_k)
 ttft_cache = ((prefix_tokens + dynamisch_invoer_tokens) / prefill_tps)

 laatste_hit_tijd = huidige_tijd
 kosten_met_cache += (k_in_cache + k_uit_basis)
 totale_ttft_met += ttft_cache

 hit_rate = (hits / aantal_verzoeken) * 100.0
 besparing_pct = ((kosten_zonder_cache - kosten_met_cache) / kosten_zonder_cache) * 100.0
 ttft_winst_pct = ((totale_ttft_zonder - totale_ttft_met) / totale_ttft_zonder) * 100.0

 print(f"Verzoeken: {aantal_verzoeken}")
 print(f"Gerealiseerde Hit Rate: {hit_rate:.2f}%")
 print(f"Kosten zonder cache: ${kosten_zonder_cache:.4f}")
 print(f"Kosten met cache: ${kosten_met_cache:.4f} (Besparing: {besparing_pct:.2f}%)")
 print(f"Gemiddelde TTFT daling: {ttft_winst_pct:.2f}%")

# Voer de simulatie uit met standaardwaarden
if __name__ == "__main__":
 random.seed(42)
 simuleer_caching_werklast()

 
## 6. Dutch-language pitfalls in prefix caching

 Designing cached prompts for Dutch-language applications raises specific linguistic challenges that directly affect the tokenization and consistency of the prefix:

 Compounds and token boundaries: Dutch is known for long compounds (such as klantenservicebeoordelingsformulier). BPE (Byte-Pair Encoding) tokenizers often break these words into varying sub-tokens depending on preceding spaces or punctuation. When a static prefix ends on an incomplete word or a dynamic injection that connects directly to a Dutch root word, the token identity of the last token block changes, causing the cache transition to fail.

 Conjugations and formal forms of address: Many applications apply dynamic prompt adjustments based on user profiles (for example switching between formal and informal address, or dynamic job titles). If these variations are injected directly into the main system message instead of into a separate, subsequent instruction layer, the cache fragments into dozens of sub-caches, each with a very low individual hit rate.

 To keep linguistic variation from undermining cache integrity, Dutch-language prompts should be built in strict layers: general system instructions and document context first, followed by style instructions (formal or informal address), and only last the specific user question.

 
## 7. A measurement protocol for validation in test and production environments

 To check whether the theoretical simulation matches reality, a structured measurement protocol is necessary. Work through the following steps before and after implementing prompt caching:

 Step 1: static separation of the prompt. Split the full prompt into a strictly static block (system prompt, document context, few-shot examples) and a dynamic block (user input, conversation history). Make sure the static part meets the provider's minimum token threshold (often at least 1024 or 2048 tokens).

 Step 2: baseline measurement without caching. Run 100 identical requests over a period of 10 minutes with caching disabled, or with random unique headers that force reuse to be bypassed. Record processing time per token, TTFT and the token costs on the invoice.

 Step 3: series measurement with caching. Activate prompt caching and repeat the 100 requests with identical prefixes at intervals of 10 seconds, 1 minute and 10 minutes respectively. Check the API response headers explicitly for the fields cached_tokens (or cache_read_input_tokens).

 Step 4: concurrency and routing stress. Send parallel requests from different client IP addresses to verify whether the model vendor's gateway maintains cache hits correctly across multiple parallel sessions.

 
## 8. Conclusions and implementation guidelines

 Prompt caching is not a guaranteed eighty percent saving but a lever that only pays off under specific architectural conditions. The main preconditions for a successful implementation are:

 First, the ratio between static input tokens and total transaction size has to be significant. Caching a 200-token system prompt yields negligible gains; caching a 10,000-token document context transforms the economic viability of an application.

 Second, traffic density has to be high enough to stay within the provider's TTL windows. For applications with low volumes or heavily fragmented contexts, writing to the cache can even lead to higher initial latency without enough cache hits to offset it.

 By running simulations in advance on historical log files, development teams can determine exactly where the tipping points lie and how prompts should be structured for maximum reuse.