6
$\begingroup$

I'm looking for a tool to track the results of several experiments/iterations in machine learning.

Training a model can take days/weeks, so it's essential to track its performance and be able to easily replicate results and roll back to previous versions. I'm looking for a tool that makes the process easy and streamlined like git did for version control.


Useful features would be:

  • open source with a license that allows commercial application
  • self-hosted [all the information should reside on computers we own]
  • python-friendly (ideally, tensorflow-friendly as well)
  • can store both hyperparameter values as well as references to specific datasets (the latter would be useful to track the effect of hard negative mining, etc.)
  • cluster-friendly
  • include basic plotting (it's useful to visualize learning curves)
  • automatic testing on multiple datasets
  • user tracking [like "blame" in git]
$\endgroup$
3
$\begingroup$

One such tool is Polyaxon. I'm using it extensively and it's really helping the experimentation pipeline.

It has features like experiment queuing, hyperparameter tuning, clustering, could be self-hosted or on the cloud, plots the metrics, it is independent of the framework and so on ...

I definitely recommend it.

| improve this answer | |
$\endgroup$
1
$\begingroup$

There is Deepkit. It's a open-source devtool for machine learning. It tracks experiments, versions code, has a model debugger, infrastructure and project management. It satisfies most of your wishes, is open-source, and crossplatform.

You can use it completely offline, self-host a server, or use the cloud server to store your experiment data.

Disclosure: I'm the author.

| improve this answer | |
$\endgroup$
1
$\begingroup$

A useful tool in this regard is Weights & Biases which covers the functionality that you describe. It is not open-source though and only free to use for personal use. It is built to do experiment tracking, and integrates well with other common tools.

The founder was a recent guest on the TWIML AI podcast. In the podcast, he explains the philosophy of the solution, and what other features might get added to their ecosystem.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.