A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from gathering data and pre-processing it, right through to post-analysis of predictions. The pipeline is essentially a large chain of modules, which can be individually examined/explained.
Here is an example image (source: DataBricks)
There is actually a nifty module (class, actually) in Scikit Learn for building your own machine learning pipelines, which is literally called Pipeline. You can specify processing steps, models and other transformations, then wrap them into a pipeline, which carries out the start-to-finish process for you. This makes it much easier to work in a modular way and alter parameters, while keeping things organised.
In the documentation, they use an ANOVA (analysis of variance) model to select variables, which are then fed into an SVM (support vector machine) to perform classification.
In the context of what might be considered a single model, a pipeline may refer to the various transformations performed on data. This might include dimensionsality reduction, embeddings, encoding/decoding (GAN example), attention mechanisims, and so on.
Here is an example of what might be referred to as a pipeline: the Spatial Transformer Network:
Images are passed through a pipeline with three parts:
- a localisation network
- a grid generator
- a sampling mechanism
these three parts might be akin to one of the parts in the MLlib Pipeline displayed above.
Another area in which pipeline is used extensively is within data management. In this case, it refers to how and where data is transferred, and perhaps by which frequency. There are large packages dedicated to building such pipelines, e.g. :
- Apache Spark - can do a lot more than just pipelines these days (use case example)
- Luigi - manages complex batch processes (developed by Spotify)
Note: this question was asked and removed just before I posted my answer, so am providing the content via a self-answered question