3
$\begingroup$

First, think it's worth me stating what I mean by replication & reproducibility:

  • Replication of analysis A results in an exact copy of all inputs and processes that are supply and result in incidental outputs in analysis B.
  • Reproducibility of analysis A results in inputs, processes, and outputs that are semantically incidental to analysis A, without access to the exact inputs and processes.

Putting aside how easy it might be to replicate a given build, especially an ad-hoc one, to me replication always possible if it's planned for and worth doing. That said, it is unclear to me is how to execute a data science workflow that allows for reproducibility.

The closet comparison I'm able to think of is documentation generators that generates software documentation intended for programmers - though the main difference I see is that in theory, if two sets of analysis ran the "reproducibility documentation generators" the documentation should match.

Another issue, is that while I get the concept of reproducibility documentation, I am having a hard time imagining what it would look like in usable form without just being a guide to replicating the analysis.

Lastly, whole intent of this is to understand if it's possible to "bake-in" reproducibility documentation as you build out a stack, not after the stack is built.

So, Is it possible to automate generating reproducibility documentation, and if so how, and what would it look like?

UPDATE: Please note that this is the second draft of this question and that Christopher Louden was kind enough to let me edit the question after I realized it was likely the first draft was unclear. Thanks!

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you give an example of a workflow which is reproducible without being a replication? $\endgroup$
    – Jay Godse
    May 14, 2014 at 21:00
  • $\begingroup$ @JayGodse: Good question, first I'd suggest reading the update I posted to the question, which might make my use of the terms a bit clearer. To answer your question, no reproduction of a workflow is a replication of the original workflow; which is to say, I don't have an answer to your question, since it is unclear to me how to you execute a data science workflow that allows for reproducibility without simply being a replication (that is, an exact copy) of what has already been done? $\endgroup$
    – blunders
    May 14, 2014 at 21:46

1 Answer 1

Reset to default
2
$\begingroup$

To be reproducible without being just a replication, you would need to redo the experiment with new data, following the same technique as before. The work flow is not as important as the techniques used. Sample data in the same way, use the same type of models. It doesn't matter if you switch from one language to another, so long as the models and the data manipulations are the same.

This type of replication will show that the results you got in the first experiment are less likely to be a fluke than they were earlier.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Right, I get that, so maybe my question is unclear. The intent of my question is how to build a workflow that allows for reproducibility. Meaning if replication requires a step-by-step guide for taking a given input and reaching a given output, how do you execute a data science workflow that allows for reproducibility without simply being a replication of what has already been done? Meaning you run the build, and it executes the code pulls the input creates the output AND generates an abstraction in text that would allow the build to be reproducible. $\endgroup$
    – blunders
    May 14, 2014 at 22:14
  • $\begingroup$ Think of it as a reproducibility documentation generator. $\endgroup$
    – blunders
    May 14, 2014 at 22:16
  • $\begingroup$ +1 So, I've thought about it, and unable to think of a way to make the question more clear without make your answer invalid, nor would it be fair for me to ask you delete your answer so I'm able to delete my question. So, just going to leave things be, and sorry my question was unclear. Cheers! $\endgroup$
    – blunders
    May 14, 2014 at 22:32
  • $\begingroup$ @blunders: please change the question. I would rather have my answer be invalid and your question answered :). $\endgroup$
    – Christopher Louden
    May 15, 2014 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.