See R's {drake}. It allows you to define a reproducible pipeline

plan <- drake_plan(
  raw_data = readxl::read_excel(file_in("raw_data.xlsx")),
  data = raw_data %>%
    mutate(Species = forcats::fct_inorder(Species)),
  hist = create_plot(data),
  fit = lm(Sepal.Width ~ Petal.Width + Species, data),
  report = rmarkdown::render(
    output_file = file_out("report.html"),
    quiet = TRUE

# call the pipeline

The great thing about drake is you that you can reload any of raw_data, data, hist, fit, report at any point. And if you change part of the code and make(plan) and {drake} will figure out which has change and just run that.

  • $\begingroup$ I see a contributor with a similar name for this project. Are you advertising your project here? Is this not against the rules? $\endgroup$
    – Valentas
    Oct 1 '19 at 8:29
  • $\begingroup$ Wow! I am asking for a Python equivalent cos I want to do the same. Also, u saw that I only made one contribution which was minor. Also the author of drake included disk.frame which was one of my package $\endgroup$
    – xiaodai
    Oct 1 '19 at 8:47
  • $\begingroup$ The question is still open. Can you accept my answer? If its not satisfactory you can comment and I can try to help you more^^ $\endgroup$ Nov 27 '19 at 8:18

Sklearn has pipeline. If you have fit and transform attributes iteratively, you can make them pipeline by Pipeline class in sklearn.pipeline.

Read the docs:


Additionally you can save and load a pipeline object by joblib.dump and joblib.load.

  • 1
    $\begingroup$ it's not the same as drake at all $\endgroup$
    – xiaodai
    Nov 27 '19 at 23:00

For larger projects snakemake is a way to go for Python (it extends Python syntax, valid Python is valid snakemake). It originates in bioinformatics and even has its own publication; it is widley adopted and used by many projects (see the literature list in the first link or the citations for the linked article).

For Jupyter notebook based projects, I made an experiment called nbpipeline which you may be interested in.


Ploomber works the same way, it keeps track of your source code and it only runs outdated steps to bring your pipeline up-to-date: https://github.com/ploomber/ploomber

Disclaimer: I'm the project's author


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.