Cianplan

Level 6: Applied: cases, FDE scenarios, mocks · L6.2 Classic cases II

Web crawler

Design the frontier, politeness and dedup.

15 min · Pro

What you take away

  • Politeness is structural: one queue per host with a delay, so a single host can never be hammered even by a bug.

And 3 more inside the class, in Pro.

How the class is laid out

  1. The prompt
  2. Requirements and the numbers
  3. The design
  4. Three decisions
  5. The twist
  6. Check your understanding