7
$\begingroup$

In the past, when trying different machine learning algorithms in order to solve a problem, I used to write drown the set of approaches on a notebook, keeping details such as features, feature preprocessing, normalization, algorithms, algorithm parameters... therefore, building a hand-written logbook.

However, currently I'm concerned about using a 'more professional' tool, so that I can keep more details and even share it with other team-members, who are also able to stamp their approaches.

It would be great an automated and collaborative tool that keep track of the work done, considering details like: features, algorithms, algorithms parameter, data pre-process, data, metrics... beyond a collaborative Google Drive Spreadsheet for instance.

How are you solving this? How are you keeping track of the work done? What's your logbook tool?

Thank you very much in advance.

$\endgroup$
4
$\begingroup$

How are you solving this? How are you keeping track of the work done? What's your logbook tool?

This might not be the best approach. But, this is how my team does it. We believe that for pulling off an end-to-end data science experiment, proper conscience is very important. So, we use Slack for the same for our discussions and the meetings.

In addition to them, we have Rmd (R markdown) files for documenting the planning and the analysis parts.

$\endgroup$
  • 1
    $\begingroup$ I like the idea of using markdown, however, I guess it's not the best approach/solution. It's good for a starter point. I also use Slack, and I highly recommend it for teams coordination and meetings. $\endgroup$ – Jorge Apr 14 '16 at 14:53
  • $\begingroup$ @Jorge Yeah, I agree that it's not the best approach :) However, I haven't found a good alternative worth spending on! $\endgroup$ – Dawny33 Apr 14 '16 at 14:58
3
$\begingroup$

Check this out, looks like exactly what you need http://www.openml.org

$\endgroup$
  • $\begingroup$ Is this a Google product? The UI looks very google-like $\endgroup$ – Dawny33 Apr 11 '16 at 11:42
  • 1
    $\begingroup$ I don't know. I am not associated with that site in any way. They have their owners and hoster information listed on the front page though. $\endgroup$ – Diego Apr 11 '16 at 11:49
2
$\begingroup$

How are you solving this? How are you keeping track of the work done? What's your logbook tool?

For my bachelors thesis (write-math.com) I wrote my own little toolkit to go through different models / preprocessing steps very fast. Each experiment had one configuration file (see hwr-experiments repository). For example:

data-source: feature-files/baseline-3-points
training: '{{nntoolkit}} train --epochs 1000 --learning-rate 0.1 --momentum 0.1 --print-errors --hook=''!detl
    test {{testing}},err=testresult_%e.txt'' {{training}} {{validation}}
    {{testing}} < {{src_model}} > {{target_model}} 2>> {{target_model}}.log'
model:
    type: mlp
    topology: 24:500:369

The trained model is stored; it is pretty fast to get the evaluation results (e.g. accuracy, confuscation matrix).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.