4
$\begingroup$

See R's {drake}. It allows you to define a reproducible pipeline

plan <- drake_plan(
  raw_data = readxl::read_excel(file_in("raw_data.xlsx")),
  data = raw_data %>%
    mutate(Species = forcats::fct_inorder(Species)),
  hist = create_plot(data),
  fit = lm(Sepal.Width ~ Petal.Width + Species, data),
  report = rmarkdown::render(
    knitr_in("report.Rmd"),
    output_file = file_out("report.html"),
    quiet = TRUE
  )
)

# call the pipeline
make(plan)

The great thing about drake is you that you can reload any of raw_data, data, hist, fit, report at any point. And if you change part of the code and make(plan) and {drake} will figure out which has change and just run that.

$\endgroup$
  • $\begingroup$ I see a contributor with a similar name for this project. Are you advertising your project here? Is this not against the rules? $\endgroup$ – Valentas Oct 1 at 8:29
  • $\begingroup$ Wow! I am asking for a Python equivalent cos I want to do the same. Also, u saw that I only made one contribution which was minor. Also the author of drake included disk.frame which was one of my package $\endgroup$ – xiaodai Oct 1 at 8:47
  • $\begingroup$ The question is still open. Can you accept my answer? If its not satisfactory you can comment and I can try to help you more^^ $\endgroup$ – Ilker Kurtulus Nov 27 at 8:18
2
$\begingroup$

Sklearn has pipeline. If you have fit and transform attributes iteratively, you can make them pipeline by Pipeline class in sklearn.pipeline.

Read the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

Additionally you can save and load a pipeline object by joblib.dump and joblib.load.

$\endgroup$
  • $\begingroup$ it's not the same as drake at all $\endgroup$ – xiaodai Nov 27 at 23:00
1
$\begingroup$

For larger projects snakemake is a way to go for Python (it extends Python syntax, valid Python is valid snakemake). It originates in bioinformatics and even has its own publication; it is widley adopted and used by many projects (see the literature list in the first link or the citations for the linked article).

For Jupyter notebook based projects, I made an experiment called nbpipeline which you may be interested in.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.