RS Glossary
Cache / KV Cache
A cache stores reusable data so a system can answer faster or avoid recomputing work, such as prompt results or model attention state.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 2
- Cache
- KV Cache
- Key-Value Cache
Remember it this wayCache the work already done; KV cache reuses prior attention work.
In plain language
A cache stores reusable results so a system does not repeat the same work. During autoregressive generation, a key-value cache keeps attention information from earlier tokens and reuses it for later tokens.
Why it matters
Caching can reduce latency and repeated computation, but it consumes memory and can preserve stale or sensitive material if its lifetime and boundaries are poorly managed.
Example
While generating a response token by token, a language model reuses the key and value tensors from the preceding tokens instead of calculating them again at every step.
Common misconception
A KV cache is not the model’s memory of a conversation in the human sense. It is an inference optimization, and it does not change the model’s learned weights.
Related terms
Sources
- Hugging Face, “Caching”explains how KV caches reuse attention keys and values during autoregressive inference.
Learn more
- Compare [[inference|Inference]], [[latency|Latency]], [[context-window|Context Window]], and [[tensor|Tensor]] to place Cache / KV Cache in the surrounding system.
- Follow the cited source material above for the method, standard, or research context behind this draft.
Public history
23 recorded events · 23 public summaries
A potential shared concept was discussed and selected for initial glossary drafting.
The captured candidate received a complete initial draft and entered section-level human review.
Radar reviewed the initial definition draft for human consideration.
Radar reviewed the initial plain-language-explanation draft for human consideration.
Radar reviewed the initial why-it-matters draft for human consideration.
Radar reviewed the initial example draft for human consideration.
Radar reviewed the initial common-misconception draft for human consideration.
Radar reviewed the initial remember-it-this-way draft for human consideration.
Radar reviewed the initial related-terms draft for human consideration.
Radar reviewed the initial sources draft for human consideration.
Radar reviewed the initial learn-more draft for human consideration.
Definition review recorded as approve.
Plain-Language Explanation review recorded as approve.
Why It Matters review recorded as approve.
Example review recorded as approve.
Common Misconception review recorded as approve.
Remember It This Way review recorded as approve.
Related Terms review recorded as approve.
Sources review recorded as approve.
Learn More review recorded as approve.
Sources review recorded as replace.
Radar verified the learner-facing sources after personal origin provenance was removed.
All nine current wording approvals were reconciled to Approved term status.
Source: vault/Glossary/terms/cache-kv-cache/term.md · sha256:8cb62f3f09a1e627b36c31524d466c18e81ac98258901d223a2fbb5a4b91192a · Release: rs-glossary-glossary-team-poc-0.5.5-2026-09-02