People often refer to pipelines when talking about models, data and even layers in a neural network. What can be meant by a pipeline?


3 Answers 3


A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from gathering data and pre-processing it, right through to post-analysis of predictions. The pipeline is essentially a large chain of modules, which can be individually examined/explained. Here is an example image (source: DataBricks)

deep learning pipeline

There is actually a nifty module (class, actually) in Scikit Learn for building your own machine learning pipelines, which is literally called Pipeline. You can specify processing steps, models and other transformations, then wrap them into a pipeline, which carries out the start-to-finish process for you. This makes it much easier to work in a modular way and alter parameters, while keeping things organised.

In the documentation, they use an ANOVA (analysis of variance) model to select variables, which are then fed into an SVM (support vector machine) to perform classification.

In the context of what might be considered a single model, a pipeline may refer to the various transformations performed on data. This might include dimensionsality reduction, embeddings, encoding/decoding (GAN example), attention mechanisims, and so on.

Here is an example of what might be referred to as a pipeline: the Spatial Transformer Network:


Images are passed through a pipeline with three parts:

  1. a localisation network
  2. a grid generator
  3. a sampling mechanism

these three parts might be akin to one of the parts in the MLlib Pipeline displayed above.

Another area in which pipeline is used extensively is within data management. In this case, it refers to how and where data is transferred, and perhaps by which frequency. There are large packages dedicated to building such pipelines, e.g. :

  1. Apache Spark - can do a lot more than just pipelines these days (use case example)
  2. Luigi - manages complex batch processes (developed by Spotify)

A pipeline could be part of a CI/Continuous Delivery/Continuous Deployment pipeline, or some kind of ETL work, like data load/extract pipeline.The idea is the automation of a group of processes that operate on the data science project you work on.Generally you draw a flowchart, there are such tools, you divide-n-conquer your entire work and set up automation tasks. It is like a flowing belt in manufacturing factory or amazon packaging system or oil pipeline. Normally you can walk through all steps manually, a kind of verification, then you can automate. In my understanding the distinction is based on what you process, either historical data, real-time data or operational logs. Kaggle is composed of a kernel, python environment, jupyter notebook and data science runs; sounds like a CI pipeline, all integrated and reproducible. I am not still comfortable with labelling though.


Often your model pipeline will rely on data generated by Data Engineers. So beyond having the ML pipeline (feature engineering, training and prediction pipeline) you also have Data engineering pipeline.

A good example can be found here https://flypipe.github.io/flypipe/html/release/1.0.1/index.html.

In my point of view as a Machine Learning, I prefer to have these pipelines implemented and linked end-to-end.

Decoupling Data and model pipelines can cause problems when serving your models behind an API that will receive raw data and therefore needs the data transformations (from data pipeline) + the features transformations (from model pipeline)

Have a look on Flypipe docs, it has a good explanation of pipelines and quick and easy to use in local python pandas or pyspark and databricks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.