The Inner Workings of Robots, Spiders, and Web Crawlers Getting Up to Speed with Webbots Lee Underwood
There are three basic types of search engines: crawler-based, human-powered, and a combination of the two.
The human-powered search engines – directories – don't really search. They rely on input from other humans. A Web site URL is submitted manually to the directory, sometimes with a short summary of the site. It's reviewed (though not always) by a human and then indexed. Directory-based search engines include Yahoo, Open Directory, and LookSmart.
The indexes of crawler-based search engines are fed data by computer programs called robots. They are also called spiders or Web crawlers because they 'crawl' over the Web. Google, AllTheWeb, and Teoma are all crawler-based search engines.
Search Engine Robots
Search Engine
User-Agent
AltaVista
Scooter
AOL
Slurp
Excite
ArchitextSpider
Google
Googlebot
Infoseek
Infoseek
Lycos
Lycos
Netscape
Googlebot
NorthernLight
Gulliver
Teoma
TeomaAgent
WebCrawler
ArchitextSpider
According to The Web Robots FAQ, "A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced." The robot doesn't physically visit the site, such as a virus would do. It accesses it by requesting documents, much like a Web browser.
There are hundreds, if not thousands, of robots sweeping across the Internet 24/7/365. Their goal is the same: track down information and return it to the
sender. Most robots are useful. They feed data to search engines like Google. Many of them are used to map the Internet. Some do a combination of mapping and indexing. A partial list of the robots (user agents) used by search engines is shown at right. A more complete list can be found at the Web Robots Pages.
There are also some robots that have a more sinister purpose. Robots such as EmailSiphon and Cherry Picker, for instance, are spambots that look to harvest e-mail addresses to add to their spam lists. Those robots scan Web pages looking for two things:
E-mail addresses, which are then used as targets for spam.
Hyperlinks, which the robots then follow, beginning the process again.
For the most part, robots are pretty straightforward when it comes to traversing a Web site. But robots don't think for themselves; they are programmed to operate in a precise manner. They don't understand JavaScript, frames, or Flash very well. They can't access password-protected areas. Often, robots can't properly index a site because there are too many bells and whistles. If they encounter any roadblocks, they move on. They don't try to figure out how to go around them.
When a robot visits a Web site it does one of two things:
it looks for the robots.txt file and the robots meta tag to see the "rules" that have been set forth, or
it begins to index the Web pages on your site.
Most, but not all, search engine robots follow the rules in the robots.txt file (more on that later). The robot then scans the visible text on the page, the content of various HTML tags, and the hyperlinks listed on the page. It will then analyze the information and process it according to an algorithm devised by its owner. Depending on the search engine, the information is indexed and sent to the search engine's database. Google's process, for instance, takes two steps:
Googlebot is the actual robot that traverses the Web site. It fetches the pages and sends them to the indexer.
The indexer sorts through every word on the page and stores them in a database.
If your Web site is online, robots will eventually come to visit. To expedite the process, most of the crawler-based search engines allow links to be submitted manually. It doesn't mean that you will be listed immediately or even the next day; it can take several weeks (or even months). However, it can help to speed-up the process so they will not have to go looking for you.
