Cianplan

Level 5: AI / LLM systems · L5.3 LLM inference serving & GPU capacity

Sizing a GPU fleet; self-host vs API

Size GPUs for a target load and compare with API cost (and a local Mac).

14 min · Pro

What you take away

  • Size in two passes: tokens per second decides the GPU count, and KV cache memory usually overrides it.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. Two passes, and the second one usually wins
  3. The crossover
  4. The memory bound
  5. KV cache
  6. Batching
  7. Optimisations
  8. Sizing
  9. Check your understanding