Search Engines & Spiders
Building Databases of web sites
As you learned in the previous two pages, no single search engine searches everything on the Web, and none of the major search engines index Web sites hand-selected by people. Here's why:
When you use a search engine you're really just searching that search engine's database of web sites. To build their databases, all search engines send out little programs called bots or spiders. These programs search, or crawl, the internet looking for web sites to add to their databases. The Web is just too large at this point (176,748,506 Web sites at last count) for any single search engine to have added every single site.
But let's face it: for most of your needs, any of the large search engines will likely provide more than enough results. The question isn't really about finding enough Web sites, it's about finding the good web sites. Next >>