Questions tagged [pipelines]
A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.
100
questions
7
votes
2
answers
2k
views
Possible harm in standardizing one-hot encoded features
While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)?
Standardizing ...
- 735
7
votes
1
answer
3k
views
how to pass parameters over sklearn pipeline's stages?
I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API.
So I builded a Sklearn Pipeline for that:
<...
- 241
7
votes
0
answers
3k
views
Tensorflow v1 Dataset API AttributeError with ndim
I'd like to make pipeline for optimizing Gpu and Cpu.
Dataset
It's about 10000 datapoint and 4 description variables for the regression problem.
...
- 71
6
votes
3
answers
955
views
What is the meaning of the term "pipeline" within data science?
People often refer to pipelines when talking about models, data and even layers in a neural network. What can be meant by a pipeline?
- 14.7k
6
votes
1
answer
4k
views
Difference between sklearn make_pipeline and imblearn make_pipeline
Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
- 161
6
votes
2
answers
1k
views
Model ensemble with Spark or Scikit Learn
I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators.
Here is a reproducible of what I would like my model to do with the Spark api
<...
- 519
5
votes
3
answers
673
views
R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?
See R's {drake}. It allows you to define a reproducible pipeline
...
- 630
5
votes
1
answer
336
views
Data Science Pipelines vs Common CD/CL
What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build,...
- 211
4
votes
1
answer
9k
views
Why GridSearchCV returns nan?
I am using gridsearchcv to tune the parameters of my model and I also use pipeline and cross-validation. When I run the model to tune the parameter of XGBoost, it returns ...
- 201
4
votes
1
answer
217
views
What can I do when my test and validation scores are good, but the submission is terrible?
This is a very broad question, I understand and I'm totally fine if someone believes it's not appropriate to do it. But it's killing me not to understand this...
Here's the thing, I'm doing a machine ...
- 141
4
votes
1
answer
2k
views
How to apply dataset balancing techniques whilst using Pipeline in Sklearn?
I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's ...
- 193
4
votes
2
answers
574
views
Scikit-learn pipeline with scaling, dimensionality reduction, average prediction of multiple regression models, and grid search cross validation
I would like to use a sklearn pipeline doing this :
( - ) scale the data ( StandardScaler )
( - ) reduce dimensionality ( PCA )
( - ) make a prediction with GradientBoostingRegressor() and ...
2
votes
1
answer
2k
views
Custom functions and pipelines
I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines.
Situation: I want to fill some missing values with the mean but using groups based on ...
- 33
2
votes
2
answers
56
views
I cannot work out the benefits of a pipeline over a linear sequence of code instructions
I've looked at quite a number of how to 'create a pipeline' instructions, but I have yet to see an explanation of the benefits over what I am showing below.
To keep my example code agnostic I'll use ...
- 121
2
votes
1
answer
440
views
TPOT machine learning
I trained a regression TPOT algorithm on Google Colab, where the output of the TPOT process is some boiler plate Python code as shown below.
...
- 403
2
votes
2
answers
668
views
Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?
I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.
I am pretty comfortable with the sklearn Pipeline ...
- 156
2
votes
1
answer
2k
views
2
votes
1
answer
4k
views
How can I get the dataframe after scikit pipeline?
I'm making many data transformations and fitting a model using scikit pipeline, but I need to extract X_train and X_test right after transformations (imputer, encoding, etc) in order to use it for ...
- 31
2
votes
1
answer
590
views
How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?
I want to apply pd.cut as a transformer in a pipeline, like this:
...
- 201
2
votes
1
answer
2k
views
sklearn - SimpleImputer in an empty Pipeline
When building a Pipeline I'm ending up at a scenario that can be simplified like this:
FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps))
Since this is one intermediary step in a ...
- 121
2
votes
1
answer
75
views
Logic Check: Building a SKLearn Pipeline
I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set.
Background:
I have a ...
- 179
2
votes
1
answer
368
views
Is it possible to change pandas column data type within a sklearn pipeline?
Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables.
Basically I need squeeze in a:
<...
- 123
2
votes
1
answer
653
views
How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?
As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and ...
- 135
2
votes
1
answer
843
views
How to restrict the columns to be passed to final classifier in PMML Pipeline
I am working on building XGBoost PMML using SKLearn and SKLearn2PMML.
I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i ...
- 141
2
votes
0
answers
321
views
PyTorch equivalent of tf.Data
I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
- 410
2
votes
0
answers
19
views
Tools/tutorials for compiling corpora for NLP experiments? [closed]
I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
- 477
1
vote
2
answers
154
views
Sklean pipeline order
Is there a correct order I should put data transformations into a pipeline using Sklearn?
Currently I have these items in my pipeline;
Feature selection, skew removal, scaling, outlier removal, ...
- 245
1
vote
1
answer
452
views
Combining sklearn pipelines with different output shape
As part of a data preprocessing step, I'm trying to create a "master pipeline" from two separate pipelines, one for numerical features and one for datetime features. The numerical pipeline ...
- 23
1
vote
1
answer
481
views
ColumnTransformer worse performance than sklearn pipeline
I have an (unbalanced , binary data) pipeline model consisting of two pipelines (preprocessing and the actual model). Now I wanted to include SimpleImputer into my ...
- 183
1
vote
2
answers
2k
views
sklearn FeatureUnion vs ColumnTransformer
I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
- 227
1
vote
1
answer
244
views
How to combine preprocessor/estimator selection with hyperparameter tuning using sklearn pipelines?
I'm aware of how to use sklearn.pipeline.Pipeline() for simple and slightly more complicated use cases alike. I know how to set up pipelines for homogeneous as well ...
- 289
1
vote
2
answers
2k
views
sklearn - How to create a sequential pipeline
Update: The examples in this post were updated
I am reposting this question here after not getting a clear answer in a previous SO post
I am looking for a help building a data preprocessing pipleline ...
- 735
1
vote
1
answer
687
views
scikit-learn pipeline strategy with both numeric and categorical columns
We are implementing an sklearn pipeline as follows (pseudo code):
...
- 111
1
vote
1
answer
19
views
How to solve mismatched code dependency issues between two or more different ML models/structures/frameworks?
I'm fairly new to machine learning. Currently I am trying to build a pipeline that uses two established NLP models. One is BERT that is fairly easy to load and quite structured. Another is FLAIR which ...
- 11
1
vote
1
answer
138
views
Alternative to EC2 for running ML batch training jobs on AWS
We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training.
Most the the pipeline is on Lambda, but Lambda is known to ...
- 780
1
vote
1
answer
680
views
How to use SMOTE in Stacking in SKLearn?
I have a data set X,y and split them to train and test data.
...
- 201
1
vote
1
answer
262
views
cost-complexity-pruning-path with pipeline
I'm using Kaggle's titanic set. I'm using pieplines and I'm trying to prune my decision tree and for that I want the cost_complexity_pruning_path. The last line of code produces the error:
...
- 113
1
vote
1
answer
1k
views
Got some troubles with using OneHotEncoder to multiple categories
I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book).
...
- 11
1
vote
1
answer
541
views
How does imblearn apply the transformations during prediction?
Let's say I have a sklearn pipeline that:
Imputes the data
Randomly oversamples the minority class
...
- 111
1
vote
1
answer
133
views
Pipeline heterogeneous data
I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find.
I'm playing with the titanic dataset and I want to work with pipelines (In order ...
- 33
1
vote
1
answer
414
views
How to deal with evaluations in a multi step ML pipeline
Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data.
Ok, ...
- 111
1
vote
0
answers
66
views
How do SciKit-Learn Pipelines pass Data between Steps?
I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline.
I'd like to pass more than just a Dataframe between the transformer ...
- 617
1
vote
2
answers
534
views
Deep Neural Network Model in sklearn Pipeline
Is it possible to add a deep neural network model as the estimator/model in an sklearn Pipeline? or is it only possible for ML models as the estimator.
For example, ...
- 258
1
vote
1
answer
47
views
when I only give command 'fit', my class does 'transform' too
I have created 2 classes, first of which is:
...
- 13
1
vote
1
answer
2k
views
How to get column names after One Hot Encoding when using Pipelines?
I am using Pipeline and ColumnTransformer to preprocess the data. Basically I am using them to impute null values, scale the ...
- 1,886
1
vote
2
answers
106
views
unsupervised anomaly detection for univariate fast frequency time series data?
I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer
This time series ...
- 1,824
1
vote
0
answers
127
views
Data Pipeline Best Practices
I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi for a data pipeline. I followed the tutorial ...
- 11
1
vote
0
answers
24
views
Working with others: Tidy data vs Pretty data [closed]
When working with someone whose background and skill level in data work may not be strong, how do you best make the argument for tidy data over "pretty" data?
There are notes of what I want ...
- 111
1
vote
0
answers
72
views
Data Lineage/Traceability in Pipelines
I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal.
Does anyone ...
- 11
1
vote
0
answers
58
views
Efficient way of adding new columns to datamart without reprocessing complete pipeline
I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
- 11