This is the term used to describe how a spider will index a web site into a search engine database. The spider will "crawl" from one web site page to the next following internal links and also external links to other web sites. Search engines likes Google will crawl billions of web site pages in a couple of days.

examples of crawlers,

• Googlebot (Google)
  
• Inktomi Slurp
  
• Alexa (IA Archiver)
  
• Jeeves
  
• WISENutbot (Looksmart)
  
• LinkWalker
  
• Walhello appie
  
• Nomad