Parallel and distributed computing

Question

What is(are) the difference(s) between parallel and distributed computing? When it comes to scalability and efficiency, it is very common to see solutions dealing with computations in clusters of machines, and sometimes it is referred to as a parallel processing, or as distributed processing.

In a certain way, the computation seems to be always parallel, since there are things running concurrently. But is the distributed computation simply related to the use of more than one machine, or are there any further specificities that distinguishes these two kinds of processing? Wouldn't it be redundant to say, for example, that a computation is parallel AND distributed?

Answer 1

Simply set, 'parallel' means running concurrently on distinct resources (CPUs), while 'distributed' means running across distinct computers, involving issues related to networks.

Parallel computing using for instance OpenMP is not distributed, while parallel computing with Message Passing is often distributed.

Being in a 'distributed but not parallel' setting would mean under-using resources so it is seldom encountered but it is conceptually possible.

Answer 2

The terms "parallel computing" and "distributed computing" certainly have a large overlap, but can be differentiated further. Actually, you already did this in your question, by later asking about "parallel processing" and "distributed processing".

One could consider "distributed computing" as the more general term that involves "distributed processing" as well as, for example, "distributed storage". The common term, "distributed", usually refers to some sort of Message Passing over a network, between machines that are physically separated.

The term "parallel computing" is also in the process of being further defined, e.g. by explicitly differentiating between the terms "parallel" and "concurrent", where - roughly - the first one refers data parallelism and the latter to task parallelism, although there are hardly really strict and binding defintions.

So one could say that

• "distributed processing" usually (although not necessarily) means that it also is "parallel processing"
• "distributed computing" is more general, and also covers aspects that are not related to parallelism
• and obviously, "parallel computing"/"parallel processing" does not imply that it is "distributed"

Answer 3

The answers presented so far are very nice, but I was also expecting an emphasis on a particular difference between parallel and distributed processing: the code executed. Considering parallel processes, the code executed is the same, regardless of the level of parallelism (instruction, data, task). You write a single code, and it will be executed by different threads/processors, e.g., while computing matrices products, or generating permutations.

On the other hand, distributed computing involves the execution of different algorithms/programs at the same time in different processors (from one or more machines). Such computations are later merged into a intermediate/final results by using the available means of data communication/synchronization (shared memory, network). Further, distributed computing is very appealing for BigData processing, as it allows for exploiting disk parallelism (usually the bottleneck for large databases).

Finally, for the level of parallelism, it may be taken rather as a constraint on the synchronization. For example, in GPGPU, which is single-instruction multiple-data (SIMD), the parallelism occurs by having different inputs for a single instruction, each pair (data_i, instruction) being executed by a different thread. Such is the restraint that, in case of divergent branches, it is necessary to discard lots of unnecessary computations, until the threads reconverge. For CPU threads, though, they commonly diverge; yet, one may use synchronization structures to grant concurrent execution of specific sections of the code.