Editing Distributed web crawling (section)

==Implementations==
As of 2003, most modern commercial search engines use this technique. [[Google]] and [[Yahoo]] use thousands of individual computers to crawl the Web.

Newer projects are attempting to use a less structured, more ''ad hoc'' form of collaboration by enlisting volunteers to join the effort using, in many cases, their home or personal computers. [[LookSmart]] is the largest search engine to use this technique, which powers its [[Grub (search engine)|Grub distributed web-crawling project]]. Wikia (now known as [[Fandom (website)|Fandom]]) acquired Grub from LookSmart in 2007.<ref>{{Cite web |last= |date=2007-07-27 |title=Wikia Acquires Distributed Web Crawler Grub |url=https://techcrunch.com/2007/07/27/wikia-acquires-distributed-web-crawler-grub/ |access-date=2022-10-08 |website=TechCrunch |language=en-US}}</ref> 

This solution uses computers that are connected to the [[Internet]] to crawl [[URLs|Internet addresses]] in the background. Upon downloading crawled web pages, they are compressed and sent back, together with a status flag (e.g. changed, new, down, redirected) to the powerful central servers. The servers, which manage a large database, send out new URLs to clients for testing.