KV Cache Calculator

Estimate key-value cache memory

When a language model generates text, it caches the key and value vectors for every token processed so it does not recompute them, and this key-value cache can consume significant GPU memory for long contexts. This calculator estimates the cache size from the number of layers, the hidden size, the context length, the batch size and the precision. The cache stores two tensors, keys and values, for each layer, so the memory is twice the product of these factors.

For a thirty-two-layer model with a hidden size of four thousand and ninety-six, an eight-thousand-token context in 16-bit precision uses several gigabytes of cache.

Why the KV cache matters

The key-value cache grows linearly with the context length, so long-context inference can require more memory for the cache than for the model weights themselves. This is a key constraint when serving many simultaneous users, since the batch size multiplies the cache, or when supporting very long prompts. Techniques like cache quantization and grouped attention reduce this cost.

This estimate uses a standard formula and assumes the full hidden size per layer; architectures with grouped or multi-query attention use less. All calculation happens locally in your browser.

Related tools

• AI Prompt Leakage Analyzer — Paste a system prompt and a hostile user input to see whether the prompt holds secrets and whether the input carries injection patterns. Local, instant.
• LLM Data Exposure Checker — Check if text contains data likely to be memorized or exposed by LLMs.
• AI Usage Policy Generator — Generate an acceptable use policy for AI tools in your company.
• Model Hallucination Estimator — Estimate risk of hallucinations based on task type and temperature.