Editing Distributed web crawling (section)

===Dynamic assignment===
With this type of policy, a central server assigns new URLs to different crawlers dynamically. This allows the central server to, for instance, dynamically balance the load of each crawler.<ref>{{Cite book |last1=Guerriero |first1=A. |last2=Ragni |first2=F. |last3=Martines |first3=C. |title=2010 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications |chapter=A dynamic URL assignment method for parallel web crawler |date=2010 |chapter-url=https://ieeexplore.ieee.org/document/5611764 |pages=119–123 |doi=10.1109/CIMSA.2010.5611764|isbn=978-1-4244-7228-4 |s2cid=14817039 }}</ref>

With dynamic assignment, typically the systems can also add or remove downloader processes. The central server may become the bottleneck, so most of the workload must be transferred to the distributed crawling processes for large crawls.

There are two configurations of crawling architectures with dynamic assignments that have been described by Shkapenyuk and Suel:<ref>
{{cite conference
| url = http://cis.poly.edu/tr/tr-cis-2001-03.htm
| accessdate =2015-10-13
| title =Design and implementation of a high-performance distributed web crawler
|author1=Shkapenyuk, Vladislav  |author2=Suel, Torsten
| year =2002
| book-title =Data Engineering, 2002. Proceedings. 18th International Conference on
| publisher =IEEE
| pages =357–368
}}
</ref>
* A small crawler configuration, in which there is a central [[Domain Name System|DNS]] resolver and central queues per Web site, and distributed downloaders.
* A large crawler configuration, in which the DNS resolver and the queues are also distributed.