Editing Load balancing (computing) (section)

====Shared and distributed memory====
Parallel computers are often divided into two broad categories: those where all processors share a single common memory on which they read and write in parallel ([[Parallel random-access machine|PRAM]] model), and those where each computing unit has its own memory ([[distributed memory]] model), and where information is exchanged by messages.

For [[shared-memory]] computers, managing write conflicts greatly slows down the speed of individual execution of each computing unit. However, they can work perfectly well in parallel. Conversely, in the case of message exchange, each of the processors can work at full speed. On the other hand, when it comes to collective message exchange, all processors are forced to wait for the slowest processors to start the communication phase.
 
In reality, few systems fall into exactly one of the categories. In general, the processors each have an internal memory to store the data needed for the next calculations and are organized in successive [[Computer cluster|clusters]]. Often, these processing elements are then coordinated through [[distributed memory]] and [[message passing]]. Therefore, the load balancing algorithm should be uniquely adapted to a parallel architecture. Otherwise, there is a risk that the efficiency of parallel problem solving will be greatly reduced.