Editing Web crawler (section)

====URL normalization====
{{Main|URL normalization}}

Crawlers usually perform some type of [[URL normalization]] in order to avoid crawling the same resource more than once. The term ''URL normalization'', also called ''URL canonicalization'', refers to the process of modifying and standardizing a URL in a consistent manner. There are several types of normalization that may be performed including conversion of URLs to lowercase, removal of "." and ".." segments, and adding trailing slashes to the non-empty path component.<ref>{{cite book |first1 = Gautam |last1 = Pant |first2 = Padmini |last2 = Srinivasan |first3 = Filippo |last3 = Menczer |editor-last = Levene |editor-first = Mark |editor2-last = Poulovassilis |editor2-first = Alexandra |editor2-link = Alexandra Poulovassilis |contribution = Crawling the Web |contribution-url = http://dollar.biz.uiowa.edu/~pant/Papers/crawling.pdf |title = Web Dynamics: Adapting to Change in Content, Size, Topology and Use |year = 2004 |pages = 153–178 |publisher = Springer |isbn = 978-3-540-40676-1 |access-date = 9 May 2006 |archive-date = 20 March 2009 |archive-url = https://web.archive.org/web/20090320030832/http://dollar.biz.uiowa.edu/~pant/Papers/crawling.pdf |url-status = dead }}</ref>