7
$\begingroup$

One of the problems I often encounter is that of poor data provenance.

When I do research I continuously make modifications to my code and rerun experiments. Each time I'm faced with a number of questions, such as: do I save the old results somewhere, just in case? Should I include the parameter settings in the output filenames or perhaps save them in a different file? How do I know which version of the script was used to produce the results?

I've recently stumbled upon Sumatra, a pretty lightweight Python package that can capture Code, Data, Environment (CDE) information that can be used to track data provenance. I like the fact that it can be used both from the command line and from within my Python scripts and requiring no GUI. The downside is that the project seems inactive and perhaps there's something better out there.

My question is: what is a good lightweight data provenance solution for my research? I'm coding small projects mostly in Python in the terminal on a remote server over SSH, so a command line solution would be perfect for me.

EDIT 2019: The Sumatra project seems inactive and other, more mature project have come along. DVC looks very promising and I've been in contact some of its authors, who have proven very helpful and supportive.

Improve this question
$\endgroup$
1
  • $\begingroup$ You have to distinguish between approaches, frameworks and tools. Reproducible research approach (paradigm) is IMHO the key. Check my relevant comprehensive answer here. $\endgroup$
    – Aleksandr Blekh
    May 10, 2015 at 6:43

1 Answer 1

Reset to default
4
$\begingroup$

Yes, you should save result files before you make major mods to the code. Disk space is cheap, so you're unlikely to run into issues unless your results sets are prolific. I would suggest storing old results sets with folder names that include a time stamp of when they were generated.

As far as time shots of your code, using github (or some other code repository tool) is as easy as can be, will save version information, allows for collaboration, and is an all around great way to backup and version your code.

The combination of these two things, you'll effectively have an easy way of mapping a result set to a specific version of the code.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your answer. These are indeed very good tips, but I think that some automation here would really help me here. I've stuck with Sumatra (see edit of my question) which forces me to make commits before I run experiments and keeps very detailed records. $\endgroup$
    – Benjamin B.
    May 8, 2015 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.