Editing Single program, multiple data (section)

==Distributed memory==
On [[distributed memory]] computer architectures, SPMD implementations usually employ [[message passing]] programming. A distributed memory computer consists of a collection of interconnected, independent computers, called nodes. For parallel execution, each node starts its own program and communicates with other nodes by sending and receiving messages, calling send/receive routines for that purpose. Other ''parallelization directives'' such as [[Barrier (computer science)|Barrier]] [[synchronization]]   may also be implemented by messages. The messages can be sent by a number of communication mechanisms, such as [[TCP/IP]] over [[Ethernet]], or specialized high-speed interconnects such as [[Myrinet]] and Supercomputer Interconnect. For distributed memory environments, serial sections of the program can be implemented by identical computation of the serial section on all nodes rather than computing the result on one node and sending it to the others, if that improves performance by reducing communication overhead.

Nowadays, the programmer is isolated from the details of the message passing by standard interfaces, such as [[PVM]] and [[Message Passing Interface|MPI]].

Distributed memory is the programming style used on parallel supercomputers from homegrown [[Beowulf cluster]]s to the largest clusters on the [[Teragrid]], as well as present [[Graphics processing unit|GPU]]-based supercomputers.