Cianplan

Level 5: AI / LLM systems · L5.2 RAG & vector databases

Ingestion and chunking

Design parsing, chunking and re-embedding for fresh documents.

13 min · Pro

What you take away

  • The chunk is the unit of retrieval, so it has to make sense on its own — a passage that needs its heading to be intelligible will be retrieved without it.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The problem
  2. The chunk is the unit
  3. Keep it derived
  4. Check your understanding