10
$\begingroup$

I am a hands on researcher and I like testing out viable solutions, so I tend to run a lot of experiments. For example, if I am calculating a similarity score between documents, I might want to try out many measures. In fact, for each measure I might need to make several runs to test the effect of some parameters.

So far, I've been tracking the runs inputs and their results by writing out the results into files with as much info about the inputs. The problem is that retrieving a specific result becomes a challenge sometimes, even if I try to add the input info to th filename. I tried using a spreadsheet with links to results but this isn't making a huge difference.

What tools/process do you use for the book keeping of your experiments?

$\endgroup$
  • 1
    $\begingroup$ I wanted to add the tag best-practices but I cannot since I don't have 150 reputation points. To be honest, I don't understand how a new comer can effectively be a contributor to the site with all such rules. I see a lot of questions for which I know the answers, but I can't answer or even up vote the answer if it is already there. $\endgroup$ – machine-wisdom Oct 5 '14 at 6:28
  • $\begingroup$ I think this is more of a generic programming question, so StackOverflow might be better. You can include a snippet of what you are trying to do, and why it's slow, and ask for suggested optimizations. $\endgroup$ – Sean Owen Oct 5 '14 at 16:46
  • $\begingroup$ Actually I think opendata.stackexchange.com would be a better fit. $\endgroup$ – Emre Oct 9 '14 at 2:36
  • 1
    $\begingroup$ I recently took a quick look around the market for products that would help with that, and I want to share my findings. There are two SAAS products that can help a Data Science team to share analyses done in Python and R. They both have an IPython notebook like IDE, and they both build around it a lot of features for running and sharing jobs. I find them both almost identical: [Domino Data Lab][1] and [Sense.io][2] [1]: dominodatalab.com [2]: sense.io $\endgroup$ – machine-wisdom Apr 24 '15 at 2:36
  • $\begingroup$ There is also a Machine Learning environment for Hadoop, which keeps track of Job runs; [h2o.ai][3]. It is not meant for being a tool for streamlining the work of the data team, but with some careful naming conventions it can help a lot. This one fits most with [3]: h2o.ai $\endgroup$ – machine-wisdom Apr 24 '15 at 2:37
4
$\begingroup$

you might want to look at http://deeplearning.net/software/jobman/intro.html

it was designed for deep learning (I guess), but it is application agnostic. It is effectively an API version of SeanEasters approach

$\endgroup$
4
$\begingroup$

I recently ran into a similar problem: How to manage extracting a variety of features from a large dataset, without knowing up front what all of them would be. (Even calculating mean values repeatedly would be computationally expensive.) Further, how would I manage predictions based on different feature sets? Meaning, if I added a new feature, how would I know which models to train on new features? It could quickly snowball into a huge mess.

My current solution is to track it all in a local NoSQL database (MongoDB). For example, I might have a collection features, each entry of which has a name, a description of how the feature was was calculated, the python file that ran the extraction, etc.

Likewise, a collection models includes models run on the data. Each entry might have a name, a list of features that were used to train the model, its eventual parameters, predicted values on a held-out test set, metrics for how the model performed, etc.

From my vantage point, this has a number of benefits:

  • By saving predictions, I can use them later in ensemble predictions.
  • Because I keep track of which features were used, I know which ones need retraining as I extract more features.
  • By saving model descriptions, I ensure that I always know what I've tried. I never have to wonder, "Have I tried LASSO with regularization parameters set by grid-search CV?" I can always look it up, and see how successful it was.

From your question, it sounds like you could adapt this approach to your problem's workflow. Install Mongo or another database of choice, and then save each experimental run, its inputs, its results, and anything else you might wish to track over the course of the project. This should be much easier to query than a spreadsheet, at the least.

$\endgroup$
  • $\begingroup$ i would like to upvote but I am not allowed. $\endgroup$ – machine-wisdom Oct 13 '14 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.