taxiongo
Member
A Spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web."