6
$\begingroup$

I'm looking for a tool to track the results of several experiments/iterations in machine learning.

Training a model can take days/weeks, so it's essential to track its performance and be able to easily replicate results and roll back to previous versions. I'm looking for a tool that makes the process easy and streamlined like git did for version control.


Useful features would be:

  • open source with a license that allows commercial application
  • self-hosted [all the information should reside on computers we own]
  • python-friendly (ideally, tensorflow-friendly as well)
  • can store both hyperparameter values as well as references to specific datasets (the latter would be useful to track the effect of hard negative mining, etc.)
  • cluster-friendly
  • include basic plotting (it's useful to visualize learning curves)
  • automatic testing on multiple datasets
  • user tracking [like "blame" in git]
$\endgroup$

3 Answers 3

3
$\begingroup$

One such tool is Polyaxon. I'm using it extensively and it's really helping the experimentation pipeline.

It has features like experiment queuing, hyperparameter tuning, clustering, could be self-hosted or on the cloud, plots the metrics, it is independent of the framework and so on ...

I definitely recommend it.

$\endgroup$
1
$\begingroup$

There is Deepkit. It's a open-source devtool for machine learning. It tracks experiments, versions code, has a model debugger, infrastructure and project management. It satisfies most of your wishes, is open-source, and crossplatform.

You can use it completely offline, self-host a server, or use the cloud server to store your experiment data.

Disclosure: I'm the author.

$\endgroup$
1
$\begingroup$

A useful tool in this regard is Weights & Biases which covers the functionality that you describe. It is not open-source though and only free to use for personal use. It is built to do experiment tracking, and integrates well with other common tools.

The founder was a recent guest on the TWIML AI podcast. In the podcast, he explains the philosophy of the solution, and what other features might get added to their ecosystem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.