I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to reduce the time spent with I/O; and with an efficient load balance policy, we can distribute tasks without much data replication overhead.
There are many studies on the literature that deal with the same problem, and each of them runs different experiments to evaluate their proposal. Some experiments are specific of the strategy presented, and some others, like weak scaling (scalability) and strong scaling (speedup), are common to all of the works.
The problem is the experiments are usually executed over entirely different infrastructures (disks, processors, # machines, network), and depending on what is being evaluated, it may raise false/unfair comparisons. For example, I may get 100% of speedup in my application running on 10 machines with Infiniband connection, whereas I could get the same or even worse results if my connection was Ethernet.
So, how can one honestly compare different experiments to point out efficiency gains?