Level 5: AI / LLM systems · L5.3 LLM inference serving & GPU capacity
The KV cache and PagedAttention
Compute KV cache size and explain paging.
13 min · Pro
What you take away
- KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element; it is the memory that grows with traffic.
And 3 more inside the class, in Pro.
How the class is laid out
- The problem
- Sizing the cache
- Paging, borrowed from operating systems
- Check your understanding