Cianplan

Level 5: AI / LLM systems · L5.3 LLM inference serving & GPU capacity

Batching: throughput vs latency

Explain continuous batching and its latency trade-off.

12 min · Pro

What you take away

  • Static batching makes every request wait for the slowest in its batch; continuous batching admits a new one whenever any sequence finishes a step.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. Static, then continuous
  3. Prefill gets in the way
  4. What to scale on
  5. Check your understanding