Cianplan

Level 5: AI / LLM systems · L5.8 Generative AI on AWS: Bedrock in practice (AIP-C01)

Evaluating, monitoring and troubleshooting on Bedrock

Set up Bedrock evaluations (programmatic, LLM-as-a-judge, human, RAG), invocation logging and CloudWatch metrics, and diagnose the common generative AI failures from the evidence.

13 min · Pro

What you take away

  • Bedrock Evaluations packages L5.6 as jobs: programmatic metrics, a judge model, a human work team, and RAG evaluation that scores retrieval and generation separately.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. The tasting panel and the camera on the pass
  3. Evaluations: L5.6 with AWS names
  4. Monitoring: the LLM span from L4.2.4, renamed
  5. Worked example
  6. Check your understanding