by Rob Sullivan

Anatomy of a search engine crawler

When you go to a search engine and perform a search many people don’t understand how those results end up there. Some people may think that sites are submitted while others know that a piece of software finds the pages. This article explains one piece of that puzzle: The search engine crawler.

Todays search engines rely on software packages called spiders or robots. These automated tools are used to search the web to discover new pages.

A brief history of search crawlers

The first crawler was the World Wide Web Wander and it appeared in 1993. It was developed by MIT and it’s initial purpose was to measure the growth of the web. Soon after, however, an index was generated from the results

Twitter This

Usuario: Password:

Technorati Tags: Anatomy, Automated Tools, Brief History, Engine Crawler, Forwards, google, Indexers, Initial Purpose, meta crawler, Meta Tags, Msn, Msnbot, Rob Sullivan, Robots, Search Crawlers, search engine crawler, search engine technology, Search Engines, SEO, Slurp, Software Packages, spider, Spiders, Visible Text, website ranking, Word Processor Documents