50
$\begingroup$

I am working on a data science project using Python. The project has several stages. Each stage comprises of taking a data set, using Python scripts, auxiliary data, configuration and parameters, and creating another data set. I store the code in git, so that part is covered. I would like to hear about:

  1. Tools for data version control.
  2. Tools enabling to reproduce stages and experiments.
  3. Protocol and suggested directory structure for such a project.
  4. Automated build/run tools.
$\endgroup$
  • 2
    $\begingroup$ Where is the question in this question? Please take a moment to review the Help Center guidelines, specifically: "If your motivation for asking the question is 'I would like to participate in a discussion about ______', then you should not be asking here." $\endgroup$ – Air Jul 18 '14 at 15:05
  • $\begingroup$ "You should only ask practical, answerable questions based on actual problems that you face." $\endgroup$ – Yuval F Jul 21 '14 at 21:04
  • $\begingroup$ This is practical, answerable and based on an actual problem in much the same way that "Tell me how to perform data science" is practical, answerable and based on an actual problem. $\endgroup$ – Air Jul 21 '14 at 21:44
  • $\begingroup$ See also: Why is the accuracy of my CNN not reproducible? $\endgroup$ – Martin Thoma Feb 20 '18 at 6:02
46
$\begingroup$

The topic of reproducible research (RR) is very popular today and, consequently, is huge, but I hope that my answer will be comprehensive enough as an answer and will provide enough information for further research, should you decide to do so.

While Python-specific tools for RR certainly exist out there, I think it makes more sense to focus on more universal tools (you never know for sure what programming languages and computing environments you will be working with in the future). Having said that, let's take a look what tools are available per your list.

1) Tools for data version control. Unless you plan to work with (very) big data, I guess, it would make sense to use the same git, which you use for source code version control. The infrastructure is already there. Even if your files are binary and big, this advice might be helpful: https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git.

2) Tools for managing RR workflows and experiments. Here's a list of most popular tools in this category, to the best of my knowledge (in the descending order of popularity):

EXAMPLE. Here's an interesting article on scientific workflows with an example of the real workflow design and data analysis, based on using Kepler and myExperiment projects: http://f1000research.com/articles/3-110/v1.

There are many RR tools that implement literate programming paradigm, exemplified by LaTeX software family. Tools that help in report generation and presentation is also a large category, where Sweave and knitr are probably the most well-known ones. Sweave is a tool, focused on R, but it can be integrated with Python-based projects, albeit with some additional effort (https://stackoverflow.com/questions/2161152/sweave-for-python). I think that knitr might be a better option, as it's modern, has extensive support by popular tools (such as RStudio) and is language-neutral (http://yihui.name/knitr/demo/engines).

3) Protocol and suggested directory structure. If I understood correctly what you implied by using term protocol (workflow), generally I think that standard RR data analysis workflow consists of the following sequential phases: data collection => data preparation (cleaning, transformation, merging, sampling) => data analysis => presentation of results (generating reports and/or presentations). Nevertheless, every workflow is project-specific and, thus, some specific tasks might require adding additional steps.

For sample directory structure, you may take a look at documentation for R package ProjectTemplate (http://projecttemplate.net), as an attempt to automate data analysis workflows and projects:

enter image description here

4) Automated build/run tools. Since my answer is focused on universal (language-neutral) RR tools, the most popular tools is make. Read the following article for some reasons to use make as the preferred RR workflow automation tool: http://bost.ocks.org/mike/make. Certainly, there are other similar tools, which either improve some aspects of make, or add some additional features. For example: ant (officially, Apache Ant: http://ant.apache.org), Maven ("next generation ant": http://maven.apache.org), rake (https://github.com/ruby/rake), Makepp (http://makepp.sourceforge.net). For a comprehensive list of such tools, see Wikipedia: http://en.wikipedia.org/wiki/List_of_build_automation_software.

$\endgroup$
  • $\begingroup$ A link about literate programming here: basically, it's about commenting the code enough so that the code becomes a standalone documentation. $\endgroup$ – gaborous Jul 26 '15 at 1:07
  • $\begingroup$ @gaborous: I am aware about the literate programming's meaning and have not included any links to the paradigm, as there are many sources for that and they are very easy to find. Nevertheless, thank you for your comment. $\endgroup$ – Aleksandr Blekh Jul 26 '15 at 1:16
  • 1
    $\begingroup$ I guessed it, that's why I added this info as a comment for the interested reader :) $\endgroup$ – gaborous Jul 26 '15 at 1:28
  • 4
    $\begingroup$ This is a very comprehensive answer, but I'm surprised that one aspect seems to be missing. Cross validation is a vital component of most DS projects and typically requires a random sample, which can make reproducibility difficult. I suggest that you briefly touch upon using the same seed for random generators in order to be able to reproduce results regardless of statistical variation. Thanks! $\endgroup$ – AN6U5 Jul 10 '16 at 21:37
  • $\begingroup$ @AN6U5: Thank you for kind words! I agree - I missed that aspect (+1). Please feel free to update my answer, adding relevant brief information on cross-validation. $\endgroup$ – Aleksandr Blekh Jul 10 '16 at 21:48
23
$\begingroup$

Since I started doing research in academia I was constantly looking for a satisfactory workflow. I think that I finally found something I am happy with:

1) Put everything under version control, e.g., Git:

For hobby research projects I use GitHub, for research at work I use the private GitLab server that is provided by our university. I also keep my datasets there.

2) I do most of my analyses along with the documentation on IPython notebooks. It is very organized (for me) to have the code, the plots, and the discussion/conclusion all in one document If I am running larger scripts, I would usually put them into separate script .py files, but I would still execute them from the IPython notebook via the %run magic to add information about the purpose, outcome, and other parameters.

I have written a small cell-magic extension for IPython and IPython notebooks, called "watermark" that I use to conveniently create time stamps and keep track of the different package versions I used and also Git hashs

For example


%watermark

29/06/2014 01:19:10

CPython 3.4.1
IPython 2.1.0

compiler   : GCC 4.2.1 (Apple Inc. build 5577)
system     : Darwin
release    : 13.2.0
machine    : x86_64
processor  : i386
CPU cores  : 2
interpreter: 64bit


%watermark -d -t

29/06/2014 01:19:11 


%watermark -v -m -p numpy,scipy

CPython 3.4.1
IPython 2.1.0

numpy 1.8.1
scipy 0.14.0

compiler   : GCC 4.2.1 (Apple Inc. build 5577)
system     : Darwin
release    : 13.2.0
machine    : x86_64
processor  : i386
CPU cores  : 2
interpreter: 64bit

For more info, see the documentation here.

$\endgroup$
  • 2
    $\begingroup$ I like the watermark magic. For those who are unaware, GitHub now offers up to 5 free private repositories for users associated with academic institutions. $\endgroup$ – bogatron Jul 17 '14 at 17:22
19
$\begingroup$

The best reproducibility tool is to make a log of your actions, something like this:

experiment/input ; expected ; observation/output ; current hypothesis and if supported or rejected
exp1 ; expected1 ; obs1 ; some fancy hypothesis, supported

This can be written down on a paper, but, if your experiments fit in a computational framework, you can use computational tools to partly or completely automate that logging process (particularly by helping you track the input datasets which can be huge, and the output figures).

A great reproducibility tool for Python with a low learning curve is of course IPython/Jupyter Notebook (don't forget the %logon and %logstart magics). Tip: to make sure your notebook is reproducible, restart the kernel and try to run all cells from top to bottom (button Run All Cells): if it works, then save everything in an archive file ("freezing"), else, notably if you need to run cells in a non linear and non sequential and non obvious fashion to avoid errors, you need to rework a bit.

Another great tool that is very recent (2015) is recipy, which is very like sumatra (see below), but made specifically for Python. I don't know if it works with Jupyter Notebooks, but I know the author frequently uses them so I guess that if it's not currently supported, it will be in the future.

Git is also awesome, and it's not tied to Python. It will help you not only to keep a history of all your experiments, code, datasets, figures, etc. but also provide you with tools to maintain (git pickaxe), collaborate (blame) and debug (git-bisect) using a scientific method of debugging (called delta debugging). Here's a story of a fictional researcher trying to make his own experiments logging system, until it ends up being a facsimile of Git.

Another general tool working with any language (with a Python API on pypi) is Sumatra, which is specifically designed to help you do replicable research (replicable aims to produce the same results given the exact same code and softwares, whereas reproducibility aims to produce the same results given any medium, which is a lot harder and time consuming and not automatable).

Here is how Sumatra works: for each experiment that you conduct through Sumatra, this software will act like a "save game state" often found in videogames. More precisely, it will will save:

  • all the parameters you provided;
  • the exact sourcecode state of your whole experimental application and config files;
  • the output/plots/results and also any file produced by your experimental application.

It will then construct a database with the timestamp and other metadatas for each of your experiments, that you can later crawl using the webGUI. Since Sumatra saved the full state of your application for a specific experiment at one specific point in time, you can restore the code that produced a specific result at any moment you want, thus you have replicable research at a low cost (except for storage if you work on huge datasets, but you can configure exceptions if you don't want to save everything everytime).

Another awesome tool is GNOME's Zeitgeist (previously coded in Python but now ported to Vala), an all-compassing action journaling system, which records everything you do and it can use machine learning to summarize for a time period you want the relationship between items based on similarity and usage patterns, eg answering questions like "What was most relevant to me, while I was working on project X, for a month last year?". Interestingly, Zim Desktop Wiki, a note-taking app similar to Evernote, has a plugin to work with Zeitgeist.

In the end, you can use either Git or Sumatra or any other software you want, they will provide you with about the same replicability power, but Sumatra is specifically tailored for scientific research so it provides a few fancy tools like a web GUI to crawl your results, while Git is more tailored towards code maintenance (but it has debugging tools like git-bisect so if your experiments involve codes, it may actually be better). Or of course you can use both!

/EDIT: dsign touched a very important point here: the replicability of your setup is as important as the replicability of your application. In other words, you should at least provide a full list of the libraries and compilers you used along with their exact versions and the details of your platform.

Personally, in scientific computing with Python, I have found that packaging an application along with the libraries is just too painful, thus I now just use an all-in-one scientific python package such as Anaconda (with the great package manager conda), and just advise users to use the same package. Another solution could be to provide a script to automatically generate a virtualenv, or to package everything using the commercial Docker application as cited by dsign or the opensource Vagrant (with for example pylearn2-in-a-box which use Vagrant to produce an easily redistributable virtual environment package).

Finally, to really ensure that you have a fully working environment everytime you need, you can make a virtual machine (see VirtualBox), and you can even save the state of the machine (snapshot) with your experiment ready to run inside. Then you can just share this virtual machine with everything included so that anyone can replicate your experiment with your exact setup. This is probably the best way to replicate a software based experiment. Containers might be a more lightweight alternative, but they do not include the whole environment, so that the replication fidelity will be less robust.

/EDIT2: Here's a great video summarizing (for debugging but this can also be applied to research) what is fundamental to do reproducible research: logging your experiments and each other steps of the scientific method, a sort of "explicit experimenting".

$\endgroup$
14
$\begingroup$

Be sure to check out docker! And in general, all the other good things that software engineering has created along decades for ensuring isolation and reproductibility.

I would like to stress that it is not enough to have just reproducible workflows, but also easy to reproduce workflows. Let me show what I mean. Suppose that your project uses Python, a database X and Scipy. Most surely you will be using a specific library to connect to your database from Python, and Scipy will be in turn using some sparse algebraic routines. This is by all means a very simple setup, but not entirely simple to setup, pun intended. If somebody wants to execute your scripts, she will have to install all the dependencies. Or worse, she might have incompatible versions of it already installed. Fixing those things takes time. It will also take time to you if you at some moment need to move your computations to a cluster, to a different cluster, or to some cloud servers.

Here is where I find docker useful. Docker is a way to formalize and compile recipes for binary environments. You can write the following in a dockerfile (I'm using here plain English instead of the Dockerfile syntax):

  • Start with a basic binary environment, like Ubuntu's
  • Install libsparse-dev
  • (Pip) Install numpy and scipy
  • Install X
  • Install libX-dev
  • (Pip) Install python-X
  • Install IPython-Notebook
  • Copy my python scripts/notebooks to my binary environment, these datafiles, and these configurations to do other miscellaneous things. To ensure reproductibility, copy them from a named url instead of a local file.
  • Maybe run IPython-Notebook.

Some of the lines will be installing things in Python using pip, since pip can do a very clean work in selecting specific package versions. Check it out too!

And that's it. If after you create your Dockerfile it can be built, then it can be built anywhere, by anybody (provided they also have access to your project-specific files, e.g. because you put them in a public url referenced from the Dockerfile). What is best, you can upload the resulting environment (called an "image") to a public or private server (called a "register") for other people to use. So, when you publish your workflow, you have both a fully reproducible recipe in the form of a Dockerfile, and an easy way for you or other people to reproduce what you do:

docker run dockerregistery.thewheezylab.org/nowyouwillbelieveme

Or if they want to poke around in your scripts and so forth:

docker run -i -t dockerregistery.thewheezylab.org/nowyouwillbelieveme /bin/bash
$\endgroup$
8
$\begingroup$

Unfortunately, I do not have enough reputation points to answer to the post by Plank, so have to answer to the whole thread - sorry about that.

I am actually the developer of the open-source Collective Knowledge Framework mentioned above. It attempts to simplify sharing of artifacts and experimental workflows as reusable and reproducible Python components with unified JSON API and JSON meta shared via GitHub. They can also be connected to predictive analytics with the same unified JSON API.

We have just released the new version V1.8.1 and provided extensive documentation so hopefully it will be easier to understand the concepts now: http://github.com/ctuning/ck/wiki

We now have many academic and industrial projects based on this framework, so you may check one of them - crowdsourcing program optimization across mobile devices provided by volunteers in a reproducible way: http://cknowledge.org/repo

We also keep track of various resources related to reproducible science here: https://github.com/ctuning/ck/wiki/Enabling-open-science

Though I am primarily focusing on making computer systems' research reproducible, I had interesting chats with colleagues from other domains and seems like they have very similar problems. So, I will be very happy if our framework can be of any help to other communities! If you have any questions or suggestions, feel free to get in touch!

$\endgroup$
7
$\begingroup$

There is an entire course that is devoted to reproducible research. https://www.coursera.org/learn/reproducible-research This course is based on R, but the underlying idea can be learnt.

One simple way is to have an Ipython notebook and keep saving every dirty work you do, be it cleaning the data, exploratory analysis or building the model.

$\endgroup$
6
$\begingroup$

I recently came across the following tool - http://github.com/ctuning/ck . It is already written in Python and seems to include what you need (my colleague is using it in the pilot project to automate image recognition).

Pros:

  1. very small, portable and customizable
  2. includes web server to distribute experiments and process them using predictive analytics
  3. has a cool usage example to crowdsource and reproduce compiler optimization - http://cknowledge.org/repo

Cons:

  1. a bit low level - you need to implement your own workflow from Python components shared via GitHub using JSON API or command line
  2. documentation is somewhat complex - I really hope that they will find time to update it soon.
$\endgroup$
6
$\begingroup$

I've created and recently released an open source tool http://dvc.org or DVC that does exactly what you are trying to reach:

  1. [Tools for data version control.] DVC works on top of Git, adds data file version control (files are stored outside of Git) and tracks the dependencies between the code and the data files. DVC automatically derives the dependency graph (DAG) for code and data.
  2. [Tools enabling to reproduce stages and experiments.] dvc repro data/scores.csv reproduces all the required steps regarding DAG.
  3. [Protocol and suggested directory structure for such a project.] DVC required a data directory (data by default) where you supposed to store all data files. However, DVC transparently moves the actual content to .cache directory and creates the symlinks (yeah, I made it to work on Windows as well). The .cache directory is not synced to Git but it could be synced through the cloud (S3 or GCP) by command dvc sync data/scores.csv (it syncs corresponded data file from cache like .cache/scores.csv_29de545)
  4. [Automated build/run tools.] See from the above.

DVC tutorial is a good starting point - "Data Version Control: iterative machine learning".

$\endgroup$
5
$\begingroup$

DISCLAIMER: I work at a company, Datmo, that creates an open-source tool to do this.

The best practice for reproducibility is the following:

1) First containerize your environment into a Docker environment by creating a Dockerfile and ensuring that all dependencies are covered in that file. I found this resource to be best (https://arxiv.org/pdf/1410.0846.pdf)

2) Once you have that you'll want to decide where you can keep track of all of the performance metrics and configurations (so that you can revisit it for future experimentation)

3) Finally, write some documentation so that a new experimenter/developer is able to revisit your code, replicate it with the environment and see where you have kept your configurations and performance metrics.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.