5
$\begingroup$

One of the problems I often encounter is that of poor data provenance.

When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of questions, such as: do I save the old results somewhere, just in case? Should I include the parameter settings in the output filenames or perhaps save them in a different file? How do I know which version of the script was used to produce the results?

I've recently stumbled upon Sumatra, a pretty lightweight Python package that can capture Code, Data, Environment (CDE) information that can be used to track data provenance. I like the fact that it can be used both from the command line and from within my Python scripts and requiring no GUI. The downside is that the project seems inactive and perhaps there's something better out there.

My question is: what is a good lightweight data provenance solution for my research? I'm coding small projects mostly in Python in the terminal on a remote server over SSH, so a command line solution would be perfect for me.

EDIT: I have stuck with Sumatra. When I posted this question I didn't look into the web interface yet, but that turns out to be a unique selling point. It displays a very detailed overview of the experiments, capturing not only the state of the data and code, but also the Python environment (package dependencies and versions!) and platform information (architecture and kernel version!).

EDIT: I've updated the subject of my question to emphasize that I'm mostly concerned about provenance.

$\endgroup$
  • $\begingroup$ You have to distinguish between approaches, frameworks and tools. Reproducible research approach (paradigm) is IMHO the key. Check my relevant comprehensive answer here. $\endgroup$ – Aleksandr Blekh May 10 '15 at 6:43
3
$\begingroup$

Yes, you should save result files before you make major mods to the code. Disk space is cheap, so you're unlikely to run into issues unless your results sets are prolific. I would suggest storing old results sets with folder names that include a time stamp of when they were generated.

As far as time shots of your code, using github (or some other code repository tool) is as easy as can be, will save version information, allows for collaboration, and is an all around great way to backup and version your code.

The combination of these two things, you'll effectively have an easy way of mapping a result set to a specific version of the code.

$\endgroup$
  • $\begingroup$ Thank you for your answer. These are indeed very good tips, but I think that some automation here would really help me here. I've stuck with Sumatra (see edit of my question) which forces me to make commits before I run experiments and keeps very detailed records. $\endgroup$ – Benjamin B. May 8 '15 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.