How to speedup message passing between computing nodes

Question

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire process as possible, I'm wondering if there any standard solutions to improve the performance of the message passing layer of my application.

What changes/improvements could I apply to my code to reduce the time spent sending messages? For what it's worth, I'm communicating up to 10GB between 9 computing nodes, and the framework I'm using is implemented with OpenMPI.

Answer 1

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be relatively marginal. Maybe as high as 2-5x, but I would be very surprised if they were any faster than that.

Under the assumption that pure I/O (writing to/reading from memory) is not your limiting factor (if it is, you're probably not going to get a lot faster), I would make a strong plug for zeromq. In the words of the creators:

We took a normal TCP socket, injected it with a mix of radioactive isotopes stolen from a secret Soviet atomic research project, bombarded it with 1950-era cosmic rays, and put it into the hands of a drug-addled comic book author with a badly-disguised fetish for bulging muscles clad in spandex. Yes, ØMQ sockets are the world-saving superheroes of the networking world.

While that may be a little dramatic, zeromq sockets in my opinion are one of the most amazing pieces of software that the world of computer networks has put together in several years. I'm not sure what you're using for your message-passing layer right now, but if you're using something traditional like rabbitmq, you're liable to see speedups of multiple orders of magnitude (personally noticed about 500x, but depends a lot of architecture)

Check out some basic benchmarks here.

Answer 2

If you expect (or find) that nodes are requesting the same data more than once, perhaps you could benefit from a caching strategy? Especially where some data is used much more often than others, so you can target only the most frequently-used information.

If the data is mutable, you also need a way to confirm that it hasn't changed since the last request that's less expensive than repeating the request.

This is further complicated if each node has its own separate cache. Depending on the nature of your system and task(s), you could consider adding a node dedicated to serving information between the processing nodes, and building a single cache on that node.

For an example of when that might be a good idea, let's suppose I retrieve some data from a remote data store over a low-bandwidth connection, and I have some task(s) requiring that data, which are distributed exclusively among local nodes. I definitely wouldn't want each node requesting information separately over that low-bandwidth connection, which another node might have previously requested. Since my local I/O is much less expensive than my I/O over the low-bandwidth connection, I might add a node between the processing nodes and the remote source that acts as an intermediate server. This node would take requests from the processing nodes, communicate with the remote data store, and cache frequently-requested data to minimize the use of that low-bandwidth connection.

The core concepts here that may be applicable to your specific case are:

• Eliminate or reduce redundant I/O;
• Take advantage of trade-offs between memory use and computation time;
• Not all I/O is created equal.