How to compare experiments run over different infrastructures

Question

I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to reduce the time spent with I/O; and with an efficient load balance policy, we can distribute tasks without much data replication overhead.

There are many studies on the literature that deal with the same problem, and each of them runs different experiments to evaluate their proposal. Some experiments are specific of the strategy presented, and some others, like weak scaling (scalability) and strong scaling (speedup), are common to all of the works.

The problem is the experiments are usually executed over entirely different infrastructures (disks, processors, # machines, network), and depending on what is being evaluated, it may raise false/unfair comparisons. For example, I may get 100% of speedup in my application running on 10 machines with Infiniband connection, whereas I could get the same or even worse results if my connection was Ethernet.

So, how can one honestly compare different experiments to point out efficiency gains?

Answer 1

This is a very good question and a common situation.

In my opinion there are three different factors that must be controlled:

• Data: There exist already different benchmarks in order to evaluate algorithms and architectures. These data must be publicly available so that everybody can contrast their approaches.
• Architecture: My suggestion is to test everything on the cloud, so that everyone can contrast their results and also there is no doubt the same machines and software is used.
• Algorithms: If you have developed a distributed algorithm, it would be great to compare your algorithm on a specific data. In this case, algorithms must not be public.

So, answering your question, if you want to compare different experiments and state to what extent your distributed algorithm outperforms others, you should try to replicate as accurate as possible the same environment (data and architecture) where the experiments were carried out.

If this is not possible, my suggestion is that you test your algorithm with public data and cloud architecture so that you become a referent as you are facilitating the comparison of future algorithms.

Answer 2

Though it's easy to say, it's better to treat the environment that changes as variables, describe/estimate your algorithm's performance base on these variables. And hopefully others will do the same. Of interest, Experiments as Research Validation -- Have We Gone too Far?.

Answer 3

The following general answer is my uneducated guess, so take it with grain of salt. Hopefully, it makes sense. I think that the best way to describe or analyze experiments (as any other systems, in general) is to build their statistical (multivariate) models and evaluate them. Depending on whether environments for your set of experiments are represented by the same model or different, I see the following approaches:

1) Single model approach. Define experiments' statistical model for all environments (dependent and independent variables, data types, assumptions, constraints). Analyze it (most likely, using regression analysis). Compare results across variables, which determine (influence) different environments.

2) Multiple models approach. The same steps as previous case, but compare results across models, corresponding to different environments.