Cianplan

Level 5: AI / LLM systems · L5.4 LLM gateways, caching & cost control

Routing, fallbacks and budgets

Design model cascades, provider fallback and per-tenant token budgets.

12 min · Pro

What you take away

  • A cascade only saves money while the escalation rate stays low, and every escalated request pays both calls in latency as well as cost.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. Junior clerk, second number, department budget
  3. Cascades: cheap first, but count the escalations
  4. Fallback: an untested path that only runs on your worst day
  5. Budgets: tokens, reserved and reconciled
  6. The whole module on one invoice
  7. The gateway
  8. Prompt caching
  9. Semantic caching
  10. Routing and fallback
  11. Budgets
  12. Check your understanding