Internet Guide Logo


Last Edit: 10/01/17

WebCrawler is a term which has two meanings - in relation to online computing.

1) It refers to software which "crawls" websites to harvest information.

2) created by Brian Pinkerton in 1994. WebCrawler was a search engine which is credited as being the first search engine to provide a "full text" search technology. What does this mean? originally, in the early 1990's, search engine's tended to only crawl and index (store information in a database) webpages based on their webpage title and meta tags. Early search engine's were incapable of indexing the "full text" of the webpage and allow users to search the "full text" of the webpage they indexed in their search results (database of stored webpage information). To store the "full text" of webpages required extensive hardware to store and serve the data; for that "day and age", the early 1990's. Therefore, when WebCrawler achieved the feat, it was an evolution for search engines: as a "full text" search tended to provide far more relevent search results. Due to WebCrawler's success, it was purchased by AoL in 1995, it was then sold to Excite in 1997, and eventually sold to Infospace in 2001 (who still own and maintain the website: 2014). However, as WebCrawler was sold, development of it's search technology ground to a halt, and the website finally morphed into a meta search engine. Which means, it does not (2014) generate it's own search results, but combines search results from a variety of third party sources, such as: Google, FAST, Ask Jeeves and Inktomi to name a few.