Internet Guide Logo

World Wide Web Wanderer

Last Edit: 10/01/17

The World Wide Web Wanderer, or simple, 'the Wanderer', was one of the earliest web crawlers. The Wanderer was created in 1993, by a student of the Massachusetts Institute of Technology, named: Matthew Gray. While studying at MIT, Gray was also a member of the Student Information Processing Board (SIPB); which setup MIT's first website: Gray left MIT in 1994, and setup a software company that developed tools for web developers. Gray is currently (2010) working for Google.

While crawlers are used by search engines to find content the World Wide Web - and index it into a database that can be queried - that was not the purpose behind creating the Wanderer. Gray did not create the Wanderer as a tool for developing a search engine: he simple wanted to research the size of the World Wide Web for academic purposes.

In 1993, when the Wanderer was created, the World Wide Web was in it's infancy and it was only just beginning to become popular. No search engines existed, and people found web content from lists of hyperlinks that were created manually. Therefore, nobody had an accurate picture of how many website existed: due to the time it takes to create and edit a manual list of websites. The only suitable option was to create a software tool that could automate the process, and this tool was a crawler.

In 1993, using the data collected by the Wanderer, gray wrote a document named: Measuring the Growth of the Web. His report highlighted the phenomenal growth of the World Wide Web and the Internet. And this document highlights the fact that he created the Wanderer to find new websites, and evaluate the growth of the World Wide Web.