Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
2
votes
1answer
53 views

how to pass parameters over sklearn pipeline's stages?

I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API. So I builded a Sklearn Pipeline for that: <...
0
votes
1answer
31 views

Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels ...
1
vote
0answers
30 views

Scalable Data Science Pipeline

I was looking at this repository (https://github.com/jeffheaton/jh-kaggle-util) and was wondering if there's a better approach to creating a scalable machine learning pipeline for data science ...
1
vote
1answer
29 views

Difference between sklearn make_pipeline and imblearn make_pipeline

Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
1
vote
1answer
23 views

How to deal with evaluations in a multi step ML pipeline

Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data. Ok, ...
1
vote
0answers
11 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
1
vote
0answers
30 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
0
votes
1answer
169 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
0
votes
0answers
14 views

Data sharing between Custom Tranformer and estimator in pyspark

Using Titanic Dataset for classification. Created custom Estimator and transformer for pipeline. some of logical operation need to be done in fit() method and data ...
1
vote
1answer
43 views

Combining scaling, dimensionality reduction, prediction using sklearn pipeline

I would like to use a sklearn pipeline doing this : ( - ) scale the data ( StandardScaler ) ( - ) reduce dimensionality ( PCA ) ( - ) make a prediction with GradientBoostingRegressor() and ...
4
votes
2answers
249 views

Model ensemble with Spark or Scikit Learn

I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators. Here is a reproducible of what I would like my model to do with the Spark api <...
0
votes
1answer
121 views

sklearn predict: IndexingError: ('Too many indexers', 'occurred at index <name>')

The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to ...
1
vote
1answer
49 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
0
votes
1answer
38 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
0answers
133 views

Which design pattern is better for data pipelines: batches or one at a time?

I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes. One ...
3
votes
2answers
263 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...