Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here

People often refer to pipelines when talking about models, data and even layers in a neural network. What can be meant by a pipeline?


A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from gathering data and pre-processing it, right through to post-analysis of predictions. The pipeline is essentially a large chain of modules, which can be individually examined/explained. Here is an example image (source: DataBricks)

deep learning pipeline

There is actually a nifty module (class, actually) in Scikit Learn for building your own machine learning pipelines, which is literally called Pipeline. You can specify processing steps, models and other transformations, then wrap them into a pipeline, which carries out the start-to-finish process for you. This makes it much easier to work in a modular way and alter parameters, while keeping things organised.

In the documentation, they use an ANOVA (analysis of variance) model to select variables, which are then fed into an SVM (support vector machine) to perform classification.

In the context of what might be considered a single model, a pipeline may refer to the various transformations performed on data. This might include dimensionsality reduction, embeddings, encoding/decoding (GAN example), attention mechanisims, and so on.

Here is an example of what might be referred to as a pipeline: the Spatial Transformer Network:


Images are passed through a pipeline with three parts:

  1. a localisation network
  2. a grid generator
  3. a sampling mechanism

these three parts might be akin to one of the parts in the MLlib Pipeline displayed above.

Another area in which pipeline is used extensively is within data management. In this case, it refers to how and where data is transferred, and perhaps by which frequency. There are large packages dedicated to building such pipelines, e.g. :

  1. Apache Spark - can do a lot more than just pipelines these days (use case example)
  2. Luigi - manages complex batch processes (developed by Spotify)

Note: this question was asked and removed just before I posted my answer, so am providing the content via a self-answered question


A pipeline could be part of a CI/Continuous Delivery/Continuous Deployment pipeline, or some kind of ETL work, like data load/extract pipeline.The idea is the automation of a group of processes that operate on the data science project you work on.Generally you draw a flowchart, there are such tools, you divide-n-conquer your entire work and set up automation tasks. It is like a flowing belt in manufacturing factory or amazon packaging system or oil pipeline. Normally you can walk through all steps manually, a kind of verification, then you can automate. In my understanding the distinction is based on what you process, either historical data, real-time data or operational logs. Kaggle is composed of a kernel, python environment, jupyter notebook and data science runs; sounds like a CI pipeline, all integrated and reproducible. I am not still comfortable with labelling though.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.