Take the 2-minute tour ×
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It's 100% free, no registration required.

I'm developing a distributed application, and as it's been designed, there'll be a great load of communication during the processing. Since the communication is already as much spread along the entire process as possible, I'm wondering if there any standard solutions to improve the performance of the message passing layer of my application.

What changes/improvements could I apply to my code to reduce the time spent sending messages? For what it's worth, I'm communicating up to 10GB between 9 computing nodes, and the framework I'm using is implemented with OpenMPI.

share|improve this question
1  
how about some compression on the messages? –  lgylym Jun 18 at 17:54
    
@user798196 That's one of the answers I was expecting. I'm just not sure about why it came as just a comment :D –  Rubens Jun 18 at 17:55
    
I would totally suggest using zmq, what are you using for your message passing layer right now? –  Slater Tyranus Jun 18 at 18:10
    
@SlaterTyranus I'm using a platform built on OpenMPI, but it does not perform message compression. Does zeromq do so? In fact, my intention here was to bring up such concepts (caching, compression, aggregation), and I was expecting them to come as answers. I'm also adding this question to a meta post, to discuss whether it is on topic. –  Rubens Jun 18 at 18:22
1  
@Rubens They considered adding it, but decided it didn't make sense for a couple reasons. Discussion here: github.com/JustinTulloss/zeromq.node/issues/285 –  Slater Tyranus Jun 18 at 18:26

2 Answers 2

up vote 4 down vote accepted

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be relatively marginal. Maybe as high as 2-5x, but I would be very surprised if they were any faster than that.

Under the assumption that pure I/O (writing to/reading from memory) is not your limiting factor (if it is, you're probably not going to get a lot faster), I would make a strong plug for zeromq. In the words of the creators:

We took a normal TCP socket, injected it with a mix of radioactive isotopes stolen from a secret Soviet atomic research project, bombarded it with 1950-era cosmic rays, and put it into the hands of a drug-addled comic book author with a badly-disguised fetish for bulging muscles clad in spandex. Yes, ØMQ sockets are the world-saving superheroes of the networking world.

While that may be a little dramatic, zeromq sockets in my opinion are one of the most amazing pieces of software that the world of computer networks has put together in several years. I'm not sure what you're using for your message-passing layer right now, but if you're using something traditional like rabbitmq, you're liable to see speedups of multiple orders of magnitude (personally noticed about 500x, but depends a lot of architecture)

Check out some basic benchmarks here.

share|improve this answer
    
That's a very nice suggestion, thanks. I'm actually using a framework implemented upon OpenMPI, and therefore I won't be able to make such changes -- although they seem to be a very promissing improvement. Btw, I very much enjoyed the citation :D –  Rubens Jun 18 at 18:50

If you expect (or find) that nodes are requesting the same data more than once, perhaps you could benefit from a caching strategy? Especially where some data is used much more often than others, so you can target only the most frequently-used information.

If the data is mutable, you also need a way to confirm that it hasn't changed since the last request that's less expensive than repeating the request.

This is further complicated if each node has its own separate cache. Depending on the nature of your system and task(s), you could consider adding a node dedicated to serving information between the processing nodes, and building a single cache on that node.

For an example of when that might be a good idea, let's suppose I retrieve some data from a remote data store over a low-bandwidth connection, and I have some task(s) requiring that data, which are distributed exclusively among local nodes. I definitely wouldn't want each node requesting information separately over that low-bandwidth connection, which another node might have previously requested. Since my local I/O is much less expensive than my I/O over the low-bandwidth connection, I might add a node between the processing nodes and the remote source that acts as an intermediate server. This node would take requests from the processing nodes, communicate with the remote data store, and cache frequently-requested data to minimize the use of that low-bandwidth connection.

The core concepts here that may be applicable to your specific case are:

  • Eliminate or reduce redundant I/O;
  • Take advantage of trade-offs between memory use and computation time;
  • Not all I/O is created equal.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.