I have some thoughts about your question. I hope it may help you to solve your problem.
I'm planning to run some experiments with very large data sets, and I'd like to distribute the computation.
In one of my posts, I have done research on topic of evaluation methods of Data Science. With Learning Curve, you can evaluate your experiments learning ability. To talk a bit more, you will fix commodity configuration, and then will run same experiment on the same number of machines with different size of data set you have (from starting from small chunk in size, incrementally increase the size until you reach the whole data set).
To point on, you should avoid having power distribution for result of performance test being run with different size of data sets. To avoid, you should carefully choose step size (step size = amount of increments).
I have about ten machines available, each with 200 GB of free space on hard disk. However, I would like to perform experiments on a greater number of nodes, to measure scalability more precisely.
For this type question, I have intuitively searched and read materials; afterwards, published as a blog post. At the end of the post, I have briefly talked about how to test your hypothesis on real complex system. If you let me, I want to briefly talk about;
First of all, base requirement should be formed in order to run data set as a whole. The minimum requirement will build your baseline evaluation score which is calculated with one/combination of evaluation metrics you have chosen, or with one/combination of methods being used to calculate
Running Time = Computation complexity + Communication cost + Synchronization cost.
After those steps, with an evaluation strategy, add new elements, e.g. new node, to the system you have doing scalability test; meanwhile, for each addition, measure performance w.r.t new system configuration.
Just to note, evaluation strategy must be planned along with considerations of default behavior of parallel and distributed systems. For example, what I mean by behavior is that adding just more cores will, after some point, automatically drop performance of the system not due to your algorithm characteristics. It is because more cores need more RAMs, more hard driver, or etc. In other words, there is a N-Way relationship between hardware components. As a second example, adding more nodes to the distributed system will punished you with more communication and synchronization costs.
As a last step, you will sketch two different graphs with your evaluation results via data analysis program or language (As a recommendation, use GNU Plot or R programming language). Print out and put those results at your desktop, and start to examine them, carefully. According to your investigation, modify/erase + rebuild evaluation strategy and re-do the performance test.
Are there commodity services which would grant me that only my application would be running at a given time? Has anyone used such services yet?
I have no much experiment on commodity services, but I can easily say whether it grants or not depends on your configuration of services. If you configured say Hadoop to your node as an only service, Hadoop will grant your code will be only running at any time.