Cianplan

Level 5: AI / LLM systems · L5.6 Evals & guardrails

How do you know it works? Error analysis first

Start evals from real failures, not generic metrics.

12 min · Pro

What you take away

  • Read forty real traces and name the failure modes before choosing a single metric; the sorted list of modes is the specification for your eval suite.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. Where learner drivers actually crash
  3. The method, in five steps
  4. Check your understanding