Editing Distributed web crawling (section)

{{Short description|Distributed computing technique}}
{{more citations needed|date=July 2008}}
'''Distributed web crawling''' is a [[distributed computing]] technique whereby [[Internet]] [[search engines]] employ many computers to [[Search engine indexing|index]] the Internet via [[web crawler|web crawling]]. Such systems may allow for users to voluntarily offer their own computing and bandwidth resources towards crawling web pages. By spreading the load of these tasks across  many computers, costs that would otherwise be spent on maintaining large computing clusters are avoided.