Level 5: AI / LLM systems · L5.3 LLM inference serving & GPU capacity
Why decode is memory-bound
Estimate tokens/s from memory bandwidth and model size.
12 min · Pro
What you take away
- Every decode step reads the whole model out of memory to produce one token, so single-stream tokens/s ≈ memory bandwidth ÷ model bytes.
And 3 more inside the class, in Pro.
How the class is laid out
- The problem
- The napkin formula
- Why "memory-bound" is precise, not a figure of speech
- Check your understanding