I am working on a data science project using Python. The project has several stages. Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and creating another data set. I store the code in git, so that part is covered. I would like to hear about:
The topic of reproducible research (RR) is very popular today and, consequently, is huge, but I hope that my answer will be comprehensive enough as an answer and will provide enough information for further research, should you decide to do so.
While Python-specific tools for RR certainly exist out there, I think it makes more sense to focus on more universal tools (you never know for sure what programming languages and computing environments you will be working with in the future). Having said that, let's take a look what tools are available per your list.
1) Tools for data version control. Unless you plan to work with (very) big data, I guess, it would make sense to use the same git, which you use for source code version control. The infrastructure is already there. Even if your files are binary and big, this advice might be helpful: https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git.
2) Tools for managing RR workflows and experiments. Here's a list of most popular tools in this category, to the best of my knowledge (in the descending order of popularity):
Taverna Workflow Management System (http://www.taverna.org.uk) - very solid, if a little too complex, set of tools. The major tool is a Java-based desktop software. However, it is compatible with online workflow repository portal myExperiment (http://www.myexperiment.org), where user can store and share their RR workflows. Web-based RR portal, fully compatible with Taverna is called Taverna Online, but it is being developed and maintained by totally different organization in Russia (referred there to as OnlineHPC: http://onlinehpc.com).
The Kepler Project (https://kepler-project.org)
VisTrails (http://vistrails.org)
Madagascar (http://www.reproducibility.org)
EXAMPLE. Here's an interesting article on scientific workflows with an example of the real workflow design and data analysis, based on using Kepler and myExperiment projects: http://f1000research.com/articles/3-110/v1.
There are many RR tools that implement literate programming paradigm, exemplified by LaTeX software family. Tools that help in report generation and presentation is also a large category, where Sweave and knitr are probably the most well-known ones. Sweave is a tool, focused on R, but it can be integrated with Python-based projects, albeit with some additional effort (https://stackoverflow.com/questions/2161152/sweave-for-python). I think that knitr might be a better option, as it's modern, has extensive support by popular tools (such as RStudio) and is language-neutral (http://yihui.name/knitr/demo/engines).
3) Protocol and suggested directory structure. If I understood correctly what you implied by using term protocol (workflow), generally I think that standard RR data analysis workflow consists of the following sequential phases: data collection => data preparation (cleaning, transformation, merging, sampling) => data analysis => presentation of results (generating reports and/or presentations). Nevertheless, every workflow is project-specific and, thus, some specific tasks might require adding additional steps.
For sample directory structure, you may take a look at documentation for R package ProjectTemplate (http://projecttemplate.net), as an attempt to automate data analysis workflows and projects:
4) Automated build/run tools. Since my answer is focused on universal (language-neutral) RR tools, the most popular tools is make. Read the following article for some reasons to use make as the preferred RR workflow automation tool: http://bost.ocks.org/mike/make. Certainly, there are other similar tools, which either improve some aspects of make, or add some additional features. For example: ant (officially, Apache Ant: http://ant.apache.org), Maven ("next generation ant": http://maven.apache.org), rake (https://github.com/ruby/rake), Makepp (http://makepp.sourceforge.net). For a comprehensive list of such tools, see Wikipedia: http://en.wikipedia.org/wiki/List_of_build_automation_software.
Since I started doing research in academia I was constantly looking for a satisfactory workflow. I think that I finally found something I am happy with:
For hobby research projects I use GitHub, for research at work I use the private GitLab server that is provided by our university. I also keep my datasets there.
I have written a small cell-magic extension for IPython and IPython notebooks, called "watermark" that I use to conveniently create time stamps and keep track of the different package versions I used and also Git hashs
For example
%watermark
29/06/2014 01:19:10
CPython 3.4.1
IPython 2.1.0
compiler : GCC 4.2.1 (Apple Inc. build 5577)
system : Darwin
release : 13.2.0
machine : x86_64
processor : i386
CPU cores : 2
interpreter: 64bit
%watermark -d -t
29/06/2014 01:19:11
%watermark -v -m -p numpy,scipy
CPython 3.4.1
IPython 2.1.0
numpy 1.8.1
scipy 0.14.0
compiler : GCC 4.2.1 (Apple Inc. build 5577)
system : Darwin
release : 13.2.0
machine : x86_64
processor : i386
CPU cores : 2
interpreter: 64bit
For more info, see the Github repository here.
The best reproducibility tool is to make a log of your actions, something like this:
experiment/input ; expected ; observation/output ; current hypothesis and if supported or rejected
exp1 ; expected1 ; obs1 ; some fancy hypothesis, supported
This can be written down on a paper, but, if your experiments fit in a computational framework, you can use computational tools to partly or completely automate that logging process (particularly by helping you track the input datasets which can be huge, and the output figures).
A great reproducibility tool for Python with a low learning curve is of course IPython/Jupyter Notebook (don't forget the %logon and %logstart magics). Tip: to make sure your notebook is reproducible, restart the kernel and try to run all cells from top to bottom (button Run All Cells): if it works, then save everything in an archive file ("freezing"), else, notably if you need to run cells in a non linear and non sequential and non obvious fashion to avoid errors, you need to rework a bit.
Another great tool that is very recent (2015) is recipy, which is very like sumatra (see below), but made specifically for Python. I don't know if it works with Jupyter Notebooks, but I know the author frequently uses them so I guess that if it's not currently supported, it will be in the future.
Git is also awesome, and it's not tied to Python. It will help you not only to keep a history of all your experiments, code, datasets, figures, etc. but also provide you with tools to maintain (git pickaxe), collaborate (blame) and debug (git-bisect) using a scientific method of debugging (called delta debugging). Here's a story of a fictional researcher trying to make his own experiments logging system, until it ends up being a facsimile of Git.
Another general tool working with any language (with a Python API on pypi) is Sumatra, which is specifically designed to help you do replicable research (replicable aims to produce the same results given the exact same code and softwares, whereas reproducibility aims to produce the same results given any medium, which is a lot harder and time consuming and not automatable).
Here is how Sumatra works: for each experiment that you conduct through Sumatra, this software will act like a "save game state" often found in videogames. More precisely, it will will save:
It will then construct a database with the timestamp and other metadatas for each of your experiments, that you can later crawl using the webGUI. Since Sumatra saved the full state of your application for a specific experiment at one specific point in time, you can restore the code that produced a specific result at any moment you want, thus you have replicable research at a low cost (except for storage if you work on huge datasets, but you can configure exceptions if you don't want to save everything everytime).
Another awesome tool is GNOME's Zeitgeist (previously coded in Python but now ported to Vala), an all-compassing action journaling system, which records everything you do and it can use machine learning to summarize for a time period you want the relationship between items based on similarity and usage patterns, eg answering questions like "What was most relevant to me, while I was working on project X, for a month last year?". Interestingly, Zim Desktop Wiki, a note-taking app similar to Evernote, has a plugin to work with Zeitgeist.
In the end, you can use either Git or Sumatra or any other software you want, they will provide you with about the same replicability power, but Sumatra is specifically tailored for scientific research so it provides a few fancy tools like a web GUI to crawl your results, while Git is more tailored towards code maintenance (but it has debugging tools like git-bisect so if your experiments involve codes, it may actually be better). Or of course you can use both!
/EDIT: dsign touched a very important point here: the replicability of your setup is as important as the replicability of your application. In other words, you should at least provide a full list of the libraries and compilers you used along with their exact versions and the details of your platform.
Personally, in scientific computing with Python, I have found that packaging an application along with the libraries is just too painful, thus I now just use an all-in-one scientific python package such as Anaconda (with the great package manager conda), and just advise users to use the same package. Another solution could be to provide a script to automatically generate a virtualenv, or to package everything using the commercial Docker application as cited by dsign or the opensource Vagrant (with for example pylearn2-in-a-box which use Vagrant to produce an easily redistributable virtual environment package).
Finally, to really ensure that you have a fully working environment everytime you need, you can make a virtual machine (see VirtualBox), and you can even save the state of the machine (snapshot) with your experiment ready to run inside. Then you can just share this virtual machine with everything included so that anyone can replicate your experiment with your exact setup. This is probably the best way to replicate a software based experiment. Containers might be a more lightweight alternative, but they do not include the whole environment, so that the replication fidelity will be less robust.
/EDIT2: Here's a great video summarizing (for debugging but this can also be applied to research) what is fundamental to do reproducible research: logging your experiments and each other steps of the scientific method, a sort of "explicit experimenting".
Be sure to check out docker! And in general, all the other good things that software engineering has created along decades for ensuring isolation and reproductibility.
I would like to stress that it is not enough to have just reproducible workflows, but also easy to reproduce workflows. Let me show what I mean. Suppose that your project uses Python, a database X and Scipy. Most surely you will be using a specific library to connect to your database from Python, and Scipy will be in turn using some sparse algebraic routines. This is by all means a very simple setup, but not entirely simple to setup, pun intended. If somebody wants to execute your scripts, she will have to install all the dependencies. Or worse, she might have incompatible versions of it already installed. Fixing those things takes time. It will also take time to you if you at some moment need to move your computations to a cluster, to a different cluster, or to some cloud servers.
Here is where I find docker useful. Docker is a way to formalize and compile recipes for binary environments. You can write the following in a dockerfile (I'm using here plain English instead of the Dockerfile syntax):
Some of the lines will be installing things in Python using pip, since pip can do a very clean work in selecting specific package versions. Check it out too!
And that's it. If after you create your Dockerfile it can be built, then it can be built anywhere, by anybody (provided they also have access to your project-specific files, e.g. because you put them in a public url referenced from the Dockerfile). What is best, you can upload the resulting environment (called an "image") to a public or private server (called a "register") for other people to use. So, when you publish your workflow, you have both a fully reproducible recipe in the form of a Dockerfile, and an easy way for you or other people to reproduce what you do:
docker run dockerregistery.thewheezylab.org/nowyouwillbelieveme
Or if they want to poke around in your scripts and so forth:
docker run -i -t dockerregistery.thewheezylab.org/nowyouwillbelieveme /bin/bash
Unfortunately, I do not have enough reputation points to answer to the post by Plank, so have to answer to the whole thread - sorry about that.
I am actually the developer of the open-source Collective Knowledge Framework mentioned above. It attempts to simplify sharing of artifacts and experimental workflows as reusable and reproducible Python components with unified JSON API and JSON meta shared via GitHub. They can also be connected to predictive analytics with the same unified JSON API.
We have just released the new version V1.8.1 and provided extensive documentation so hopefully it will be easier to understand the concepts now: http://github.com/ctuning/ck/wiki
We now have many academic and industrial projects based on this framework, so you may check one of them - crowdsourcing program optimization across mobile devices provided by volunteers in a reproducible way: http://cknowledge.org/repo
We also keep track of various resources related to reproducible science here: https://github.com/ctuning/ck/wiki/Enabling-open-science
Though I am primarily focusing on making computer systems' research reproducible, I had interesting chats with colleagues from other domains and seems like they have very similar problems. So, I will be very happy if our framework can be of any help to other communities! If you have any questions or suggestions, feel free to get in touch!
There is an entire course that is devoted to reproducible research. https://www.coursera.org/learn/reproducible-research This course is based on R, but the underlying idea can be learnt.
One simple way is to have an Ipython notebook and keep saving every dirty work you do, be it cleaning the data, exploratory analysis or building the model.
I recently came across the following tool - http://github.com/ctuning/ck . It is already written in Python and seems to include what you need (my colleague is using it in the pilot project to automate image recognition).
Pros:
Cons:
I've created and recently released an open source tool http://dvc.org or DVC that does exactly what you are trying to reach:
dvc repro data/scores.csv reproduces all the required steps regarding DAG.data by default) where you supposed to store all data files. However, DVC transparently moves the actual content to .cache directory and creates the symlinks (yeah, I made it to work on Windows as well). The .cache directory is not synced to Git but it could be synced through the cloud (S3 or GCP) by command dvc sync data/scores.csv (it syncs corresponded data file from cache like .cache/scores.csv_29de545)DVC tutorial is a good starting point - "Data Version Control: iterative machine learning".
DISCLAIMER: I work at a company, Datmo, that creates an open-source tool to do this.
The best practice for reproducibility is the following:
1) First containerize your environment into a Docker environment by creating a Dockerfile and ensuring that all dependencies are covered in that file. I found this resource to be best (https://arxiv.org/pdf/1410.0846.pdf)
2) Once you have that you'll want to decide where you can keep track of all of the performance metrics and configurations (so that you can revisit it for future experimentation)
3) Finally, write some documentation so that a new experimenter/developer is able to revisit your code, replicate it with the environment and see where you have kept your configurations and performance metrics.
