Cianplan

Level 5: AI / LLM systems · L5.8 Generative AI on AWS: Bedrock in practice (AIP-C01)

Fairness and drift in foundation model apps: evaluations, guardrails, logs and A/B tests

Measure stereotyping and harm in a foundation model application with evaluation jobs, keep guardrails (runtime control) and evaluations (measurement) in their separate roles, detect drift from invocation logs and scheduled evaluations, and compare prompt or flow versions with controlled A/B tests.

13 min · Pro

What you take away

  • AWS frames guardrails as continuous runtime protection and evaluations as measurement; a guardrail blocks, it does not tell you how biased the outputs are.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. Tasting, the bouncer and the camera
  3. Measure with evaluations, control with guardrails
  4. Drift without a weight changing
  5. A/B tests and the evidence trail
  6. Check your understanding