Editing Parallel computing (section)

====Distributed computing====
{{main|Distributed computing}}
A distributed computer (also known as a distributed memory multiprocessor) is a distributed memory computer system in which the processing elements are connected by a network. Distributed computers are highly scalable. The terms "[[concurrent computing]]", "parallel computing", and "distributed computing" have a lot of overlap, and no clear distinction exists between them.<ref>[[Distributed computing#CITEREFGhosh2007|Ghosh (2007)]], p. 10. [[Distributed computing#CITEREFKeidar2008|Keidar (2008)]].</ref> The same system may be characterized both as "parallel" and "distributed"; the processors in a typical distributed system run concurrently in parallel.<ref>[[Distributed computing#CITEREFLynch1996|Lynch (1996)]], p. xix, 1–2. [[Distributed computing#CITEREFPeleg2000|Peleg (2000)]], p. 1.</ref>

=====Cluster computing=====
{{main|Computer cluster}}

[[File:Beowulf.jpg|right|thumbnail|upright|A [[Beowulf (computing)|Beowulf cluster]]]]

A cluster is a group of loosely coupled computers that work together closely, so that in some respects they can be regarded as a single computer.<ref>[http://www.webopedia.com/TERM/c/clustering.html What is clustering?] Webopedia computer dictionary. Retrieved on November 7, 2007.</ref> Clusters are composed of multiple standalone machines connected by a network. While machines in a cluster do not have to be symmetric, [[Load balancing (computing)|load balancing]] is more difficult if they are not. The most common type of cluster is the [[Beowulf (computing)|Beowulf cluster]], which is a cluster implemented on multiple identical [[commercial off-the-shelf]] computers connected with a [[TCP/IP]] [[Ethernet]] [[local area network]].<ref>[https://www.pcmag.com/encyclopedia_term/0,2542,t=Beowulf&i=38548,00.asp Beowulf definition.] {{Webarchive|url=https://web.archive.org/web/20121010215231/https://www.pcmag.com/encyclopedia_term/0%2C2542%2Ct%3DBeowulf%26i%3D38548%2C00.asp |date=2012-10-10 }} ''PC Magazine''. Retrieved on November 7, 2007.</ref> Beowulf technology was originally developed by [[Thomas Sterling (computing)|Thomas Sterling]] and [[Donald Becker]]. 87% of all [[TOP500|Top500]] supercomputers are clusters.<ref>{{Cite web|url=https://www.top500.org/statistics/list/|title=List Statistics {{!}} TOP500 Supercomputer Sites|website=www.top500.org|language=en|access-date=2018-08-05}}</ref> The remaining are Massively Parallel Processors, explained below.

Because grid computing systems (described below) can easily handle embarrassingly parallel problems, modern clusters are typically designed to handle more difficult problems—problems that require nodes to share intermediate results with each other more often. This requires a high bandwidth and, more importantly, a low-[[latency (engineering)|latency]] interconnection network. Many historic and current supercomputers use customized high-performance network hardware specifically designed for cluster computing, such as the Cray Gemini network.<ref>[https://www.nersc.gov/users/computational-systems/hopper/configuration/interconnect/ "Interconnect"] {{webarchive|url=https://web.archive.org/web/20150128133120/https://www.nersc.gov/users/computational-systems/hopper/configuration/interconnect/ |date=2015-01-28 }}.</ref> As of 2014, most current supercomputers use some off-the-shelf standard network hardware, often [[Myrinet]], [[InfiniBand]], or [[Gigabit Ethernet]].

=====Massively parallel computing=====
{{main|Massively parallel (computing)}}

[[File:BlueGeneL cabinet.jpg|right|thumbnail|upright|A cabinet from [[IBM]]'s [[Blue Gene|Blue Gene/L]] massively parallel [[supercomputer]]]]

A massively parallel processor (MPP) is a single computer with many networked processors. MPPs have many of the same characteristics as clusters, but MPPs have specialized interconnect networks (whereas clusters use commodity hardware for networking). MPPs also tend to be larger than clusters, typically having "far more" than 100&nbsp;processors.<ref>Hennessy and Patterson, p.&nbsp;537.</ref> In an MPP, "each CPU contains its own memory and copy of the operating system and application. Each subsystem communicates with the others via a high-speed interconnect."<ref>[https://www.pcmag.com/encyclopedia_term/0,,t=mpp&i=47310,00.asp MPP Definition.] {{Webarchive|url=https://web.archive.org/web/20130511084523/https://www.pcmag.com/encyclopedia_term/0%2C%2Ct%3Dmpp%26i%3D47310%2C00.asp |date=2013-05-11 }} ''PC Magazine''. Retrieved on November 7, 2007.</ref>

[[IBM]]'s [[Blue Gene|Blue Gene/L]], the fifth fastest [[supercomputer]] in the world according to the June 2009 [[TOP500]] ranking, is an MPP.

=====Grid computing=====
{{main|Grid computing}}
Grid computing is the most distributed form of parallel computing. It makes use of computers communicating over the [[Internet]] to work on a given problem. Because of the low bandwidth and extremely high latency available on the Internet, distributed computing typically deals only with [[embarrassingly parallel]] problems.

Most grid computing applications use [[middleware]] (software that sits between the operating system and the application to manage network resources and standardize the software interface). The most common grid computing middleware is the [[Berkeley Open Infrastructure for Network Computing]] (BOINC). Often [[volunteer computing]] software makes use of "spare cycles", performing computations at times when a computer is idling.<ref>{{cite journal|last=Kirkpatrick|first=Scott|title=COMPUTER SCIENCE: Rough Times Ahead|journal=Science|volume=299|issue=5607|pages=668–669|doi=10.1126/science.1081623|year=2003|pmid=12560537|s2cid=60622095}}</ref>

=====Cloud computing=====
{{main|Cloud computing}}
The ubiquity of Internet brought the possibility of large-scale cloud computing.