Editing Distributed web crawling (section)

==Types==
Cho<ref name="cho2002parallel"/> and Garcia-Molina studied two types of policies:

===Dynamic assignment===
With this type of policy, a central server assigns new URLs to different crawlers dynamically. This allows the central server to, for instance, dynamically balance the load of each crawler.<ref>{{Cite book |last1=Guerriero |first1=A. |last2=Ragni |first2=F. |last3=Martines |first3=C. |title=2010 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications |chapter=A dynamic URL assignment method for parallel web crawler |date=2010 |chapter-url=https://ieeexplore.ieee.org/document/5611764 |pages=119–123 |doi=10.1109/CIMSA.2010.5611764|isbn=978-1-4244-7228-4 |s2cid=14817039 }}</ref>

With dynamic assignment, typically the systems can also add or remove downloader processes. The central server may become the bottleneck, so most of the workload must be transferred to the distributed crawling processes for large crawls.

There are two configurations of crawling architectures with dynamic assignments that have been described by Shkapenyuk and Suel:<ref>
{{cite conference
| url = http://cis.poly.edu/tr/tr-cis-2001-03.htm
| accessdate =2015-10-13
| title =Design and implementation of a high-performance distributed web crawler
|author1=Shkapenyuk, Vladislav  |author2=Suel, Torsten
| year =2002
| book-title =Data Engineering, 2002. Proceedings. 18th International Conference on
| publisher =IEEE
| pages =357–368
}}
</ref>
* A small crawler configuration, in which there is a central [[Domain Name System|DNS]] resolver and central queues per Web site, and distributed downloaders.
* A large crawler configuration, in which the DNS resolver and the queues are also distributed.

===Static assignment===
With this type of policy, there is a fixed rule stated from the beginning of the crawl that defines how to assign new URLs to the crawlers.

For static assignment, a hashing function can be used to transform URLs (or, even better, complete website names) into a number that corresponds to the index of the corresponding crawling process.<ref>{{Cite book |last1=Wan |first1=Yuan |last2=Tong |first2=Hengqing |title=2008 IEEE International Conference on Networking, Sensing and Control |chapter=URL Assignment Algorithm of Crawler in Distributed System Based on Hash |date=2008 |chapter-url=http://dx.doi.org/10.1109/icnsc.2008.4525482 |journal=IEEE |pages=1632–1635 |doi=10.1109/icnsc.2008.4525482|isbn=978-1-4244-1685-1 |s2cid=39188334 }}</ref> As there are external links that will go from a Web site assigned to one crawling process to a website assigned to a different crawling process, some exchange of URLs must occur.

To reduce the overhead due to the exchange of URLs between crawling processes, the exchange should be done in batch, several URLs at a time, and the most cited URLs in the collection should be known by all crawling processes before the crawl (e.g.: using data from a previous crawl).<ref name="cho2002parallel">{{cite conference|url=http://dl.acm.org/citation.cfm?id=511464|accessdate=2015-10-13|title=Parallel crawlers|author1=Cho, Junghoo|author2=Garcia-Molina, Hector|year=2002|book-title=Proceedings of the 11th international conference on World Wide Web|publisher=ACM|pages=124–135|isbn=1-58113-449-5|doi=10.1145/511446.511464|url-access=subscription}}</ref>