Editing Apache Nutch (section)

==Scalability==
IBM Research studied the performance<ref>{{Cite web|url=http://www.cecs.uci.edu/~papers/ipdps07/pdfs/SMTPS-201-paper-1.pdf|title=Scalability of the Nutch search engine}}</ref> of Nutch/Lucene as part of its Commercial Scale Out (CSO) project.<ref>{{Cite web|url=http://weather.ou.edu/~apw/projects/cso/prov_paper.pdf|archive-url=https://web.archive.org/web/20081203064621/http://weather.ou.edu/~apw/projects/cso/prov_paper.pdf|url-status=dead|title=Base Operating System Provisioning and Bringup for a Commercial Supercomputer|archive-date=December 3, 2008}}</ref> Their findings were that a [[Scalability#HORIZONTAL-SCALING|scale-out]] system, such as Nutch/Lucene, could achieve a performance level on a cluster of blades that was not achievable on any [[Scalability#VERTICAL-SCALING|scale-up]] computer such as the [[POWER5]].

The ClueWeb09 dataset (used in e.g. [[Text Retrieval Conference|TREC]]) was gathered using Nutch, with an average speed of 755.31 documents per second.<ref>[http://boston.lti.cs.cmu.edu/crawler/crawlerstats.html The Sapphire Web Crawler - Crawl Statistics]. Boston.lti.cs.cmu.edu (2008-10-01). Retrieved on 2013-07-21.</ref>