Editing Single program, multiple data (section)

==Shared memory==
On a [[Shared memory architecture|shared memory]] machine (a computer with several interconnected [[CPU]]s that access the same memory space), the sharing can be implemented in the context of either physically shared memory or logically shared (but physically distributed) memory; in addition to the shared memory, the CPUs in the computer system can also include local (or private) memory. For either of these contexts, synchronization can be enabled with hardware enabled primitives (such as [[compare-and-swap]], or [[fetch-and-add]]. For machines that do not have such hardware support, locks can be used and data can be "exchanged" across processors (or, more generally, ''[[Process (computing)|processes]]'' or [[Thread (computer science)|threads]]) by depositing the sharable data in a shared memory area. When the hardware does not support shared memory, packing the data as a "message" is often the most efficient way to program (logically) shared memory computers with large number of processors, where the physical memory is local to processors and accessing the memory of another processor takes longer. SPMD on a shared memory machine can be implemented by standard processes (heavyweight) or threads (lightweight).

Shared memory [[multiprocessing]] (both [[symmetric multiprocessing]], SMP, and [[non-uniform memory access]], NUMA) presents the programmer with a common memory space and the possibility to parallelize execution. With the (IBM) SPMD model the cooperating processors (or processes) take different paths through the program, using parallel directives (''parallelization and synchronization directives'', which can utilize compare-and-swap and fetch-and-add operations on shared memory synchronization variables), and perform operations on data in the shared memory ("shared data"); the processors (or processes) can also have access and perform operations on data in their local memory ("private data"). In contrast, with fork-and-join approaches, the program starts executing on one processor and the execution splits in a parallel region, which is started when parallel directives are encountered; in a parallel region, the processors execute a parallel task on different data. A typical example is the parallel DO loop, where different processors work on separate parts of the arrays involved in the loop. At the end of the loop, execution is synchronized (with soft- or hard-barriers<ref name=":5" />), and processors (processes) continue to the next available section of the program to execute. The (IBM) SPMD has been implemented in the current standard interface for shared memory multiprocessing, [[OpenMP]], which uses multithreading, usually implemented by lightweight processes, called [[Thread (computer science)|threads]].