5
$\begingroup$

See R's {drake}. It allows you to define a reproducible pipeline

plan <- drake_plan(
  raw_data = readxl::read_excel(file_in("raw_data.xlsx")),
  data = raw_data %>%
    mutate(Species = forcats::fct_inorder(Species)),
  hist = create_plot(data),
  fit = lm(Sepal.Width ~ Petal.Width + Species, data),
  report = rmarkdown::render(
    knitr_in("report.Rmd"),
    output_file = file_out("report.html"),
    quiet = TRUE
  )
)

# call the pipeline
make(plan)

The great thing about drake is you that you can reload any of raw_data, data, hist, fit, report at any point. And if you change part of the code and make(plan) and {drake} will figure out which has change and just run that.

Improve this question
$\endgroup$
3
  • $\begingroup$ I see a contributor with a similar name for this project. Are you advertising your project here? Is this not against the rules? $\endgroup$
    – Valentas
    Oct 1, 2019 at 8:29
  • $\begingroup$ Wow! I am asking for a Python equivalent cos I want to do the same. Also, u saw that I only made one contribution which was minor. Also the author of drake included disk.frame which was one of my package $\endgroup$
    – xiaodai
    Oct 1, 2019 at 8:47
  • $\begingroup$ The question is still open. Can you accept my answer? If its not satisfactory you can comment and I can try to help you more^^ $\endgroup$
    – Ilker Kurtulus
    Nov 27, 2019 at 8:18

3 Answers 3

Reset to default
3
$\begingroup$

Sklearn has pipeline. If you have fit and transform attributes iteratively, you can make them pipeline by Pipeline class in sklearn.pipeline.

Read the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

Additionally you can save and load a pipeline object by joblib.dump and joblib.load.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ it's not the same as drake at all $\endgroup$
    – xiaodai
    Nov 27, 2019 at 23:00
2
$\begingroup$

For larger projects snakemake is a way to go for Python (it extends Python syntax, valid Python is valid snakemake). It originates in bioinformatics and even has its own publication; it is widley adopted and used by many projects (see the literature list in the first link or the citations for the linked article).

For Jupyter notebook based projects, I made an experiment called nbpipeline which you may be interested in.

Improve this answer
$\endgroup$
1
$\begingroup$

Ploomber works the same way, it keeps track of your source code and it only runs outdated steps to bring your pipeline up-to-date: https://github.com/ploomber/ploomber

Disclaimer: I'm the project's author

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.