A spider is used by search engines such as Google and Alta Vista to find and index web site pages. The way a spider does this is by following links on a web site to internal and external pages. You can find a simple spider's at many script download web sites to crawl and index pages on a personal web site to produce search engine for that site. However a search engine will use a custom and much more complicated spider to be able to index millions of web sites.

• Infoseek (go.com) - Sidewinder ,
  
• Altavista - Scooter,
  
• Lycos - Fast-Webcrawler
  
• Excite - Architext,
  
• WebCrawler Excites - Architext,
  
• Google - Backrub or GoogleBot
  
• Northern Light - Northern Lights Gulliver
• Yahoo - Inktomi Slurp
• Looksmart - WISENutbot
• AllTheWeb - Fast-Webcrawler