Lightweight data provenance tool

Question

One of the problems I often encounter is that of poor data provenance.

When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of questions, such as: do I save the old results somewhere, just in case? Should I include the parameter settings in the output filenames or perhaps save them in a different file? How do I know which version of the script was used to produce the results?

I've recently stumbled upon Sumatra, a pretty lightweight Python package that can capture Code, Data, Environment (CDE) information that can be used to track data provenance. I like the fact that it can be used both from the command line and from within my Python scripts and requiring no GUI. The downside is that the project seems inactive and perhaps there's something better out there.

My question is: what is a good lightweight data provenance solution for my research? I'm coding small projects mostly in Python in the terminal on a remote server over SSH, so a command line solution would be perfect for me.

EDIT 2019: The Sumatra project seems inactive and other, more mature project have come along. DVC looks very promising and I've been in contact some of its authors, who have proven very helpful and supportive.

Answer 1

Yes, you should save result files before you make major mods to the code. Disk space is cheap, so you're unlikely to run into issues unless your results sets are prolific. I would suggest storing old results sets with folder names that include a time stamp of when they were generated.

As far as time shots of your code, using github (or some other code repository tool) is as easy as can be, will save version information, allows for collaboration, and is an all around great way to backup and version your code.

The combination of these two things, you'll effectively have an easy way of mapping a result set to a specific version of the code.