Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments?

Question

I'm looking for a tool to track the results of several experiments/iterations in machine learning.

Training a model can take days/weeks, so it's essential to track its performance and be able to easily replicate results and roll back to previous versions. I'm looking for a tool that makes the process easy and streamlined like git did for version control.

---

Useful features would be:

• open source with a license that allows commercial application
• self-hosted [all the information should reside on computers we own]
• python-friendly (ideally, tensorflow-friendly as well)
• can store both hyperparameter values as well as references to specific datasets (the latter would be useful to track the effect of hard negative mining, etc.)
• cluster-friendly
• include basic plotting (it's useful to visualize learning curves)
• automatic testing on multiple datasets
• user tracking [like "blame" in git]

Answer 1

One such tool is Polyaxon. I'm using it extensively and it's really helping the experimentation pipeline.

It has features like experiment queuing, hyperparameter tuning, clustering, could be self-hosted or on the cloud, plots the metrics, it is independent of the framework and so on ...

I definitely recommend it.

Answer 2

There is Deepkit. It's a open-source devtool for machine learning. It tracks experiments, versions code, has a model debugger, infrastructure and project management. It satisfies most of your wishes, is open-source, and crossplatform.

You can use it completely offline, self-host a server, or use the cloud server to store your experiment data.

Disclosure: I'm the author.

Answer 3

A useful tool in this regard is Weights & Biases which covers the functionality that you describe. It is not open-source though and only free to use for personal use. It is built to do experiment tracking, and integrates well with other common tools.

The founder was a recent guest on the TWIML AI podcast. In the podcast, he explains the philosophy of the solution, and what other features might get added to their ecosystem.