Questions tagged [pipelines]
A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.
101
questions
0
votes
1
answer
1k
views
How to remove features from a sklearn pipeline after it has already been fitted?
Background:
I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to ...
1
vote
0
answers
66
views
Data Lineage/Traceability in Pipelines
I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal.
Does anyone ...
- 11
2
votes
2
answers
584
views
Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?
I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling.
I am pretty comfortable with the sklearn Pipeline ...
- 156
0
votes
1
answer
54
views
How is the target variable passed to the final estimator in this pipeline?
There is a pipeline like below. X is features and y is the target variable.
I would like to know how ...
0
votes
0
answers
18
views
Create a pipeline for recursive eliminartion using artifical neural nets
I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated....
0
votes
1
answer
102
views
Compare cross validation and test set results
I am having a hard time understanding the results of a cross validation test and a test run on a test set.
First I made the following pipeline:
...
1
vote
2
answers
2k
views
sklearn - How to create a sequential pipeline
Update: The examples in this post were updated
I am reposting this question here after not getting a clear answer in a previous SO post
I am looking for a help building a data preprocessing pipleline ...
- 693
0
votes
0
answers
127
views
Apply two different Sklearn classifiers to two different subsets of the same data
I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows:
described: description column POPULATED
non-described: <...
0
votes
2
answers
39
views
Selecting Transforms with sklearn pipelines
So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do :
...
user98969
7
votes
2
answers
1k
views
Possible harm in standardizing one-hot encoded features
While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)?
Standardizing ...
- 693
2
votes
1
answer
792
views
How to restrict the columns to be passed to final classifier in PMML Pipeline
I am working on building XGBoost PMML using SKLearn and SKLearn2PMML.
I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i ...
- 141
1
vote
1
answer
1k
views
Got some troubles with using OneHotEncoder to multiple categories
I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book).
...
- 11
2
votes
1
answer
544
views
How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?
I want to apply pd.cut as a transformer in a pipeline, like this:
...
- 201
0
votes
2
answers
1k
views
Error when trying to run RandomForestClassifer with Pipieline and GridSearch [closed]
I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV
I am getting an error when I fit the data. I am not sure how to fix it.
Will appreciate any help on this.
My code is:
...
- 243
1
vote
1
answer
488
views
How does imblearn apply the transformations during prediction?
Let's say I have a sklearn pipeline that:
Imputes the data
Randomly oversamples the minority class
...
- 111
0
votes
1
answer
322
views
Google DataFlow
I'm trying to build a Google dataflow pipeline through one of the posts in Medium.
https://levelup.gitconnected.com/scaling-scikit-learn-with-apache-beam-251eb6fcf75b
However, it seems like I'm ...
- 103
1
vote
0
answers
42
views
Efficient way of adding new columns to datamart without reprocessing complete pipeline
I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
- 11
2
votes
1
answer
1k
views
sklearn - SimpleImputer in an empty Pipeline
When building a Pipeline I'm ending up at a scenario that can be simplified like this:
FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps))
Since this is one intermediary step in a ...
- 121
2
votes
0
answers
312
views
PyTorch equivalent of tf.Data
I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
- 400
1
vote
1
answer
123
views
Pipeline heterogeneous data
I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find.
I'm playing with the titanic dataset and I want to work with pipelines (In order ...
- 33
1
vote
0
answers
20
views
Which statistical method to use for feature selection between numerical inputs and categorical input?
I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
- 193
2
votes
1
answer
2k
views
Custom functions and pipelines
I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines.
Situation: I want to fill some missing values with the mean but using groups based on ...
- 33
4
votes
1
answer
2k
views
How to apply dataset balancing techniques whilst using Pipeline in Sklearn?
I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's ...
- 193
2
votes
1
answer
2k
views
1
vote
0
answers
920
views
ValueError: Found input variables with inconsistent numbers of samples: [2, 44] [closed]
Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned"
Then i create a ML model using the above as one of my features.
...
- 11
2
votes
1
answer
61
views
Logic Check: Building a SKLearn Pipeline
I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set.
Background:
I have a ...
- 179
1
vote
2
answers
124
views
Sklean pipeline order
Is there a correct order I should put data transformations into a pipeline using Sklearn?
Currently I have these items in my pipeline;
Feature selection, skew removal, scaling, outlier removal, ...
- 245
0
votes
1
answer
473
views
order of features importance after make_column_transformer and pipeline
I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor.
...
- 693
5
votes
1
answer
334
views
Data Science Pipelines vs Common CD/CL
What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build,...
- 211
0
votes
0
answers
2k
views
SHAP Kernel explainer for my pipeline model
I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer.
The data contains different types of ...
- 53
2
votes
0
answers
19
views
Tools/tutorials for compiling corpora for NLP experiments? [closed]
I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
- 477
0
votes
0
answers
35
views
Different extraction pipeline for train and test
I'm trying to create a production-ready ML model.
The problem is as follows:
Training data: Database A + Plus python aggregations.
Testing data: Database B + Plus python aggregations.
Both ...
- 101
1
vote
1
answer
607
views
scikit-learn pipeline strategy with both numeric and categorical columns
We are implementing an sklearn pipeline as follows (pseudo code):
...
- 111
1
vote
1
answer
316
views
Is it possible to change pandas column data type within a sklearn pipeline?
Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables.
Basically I need squeeze in a:
<...
- 113
0
votes
1
answer
1k
views
Error encoding categorical features using sklearn pipelines
I am new to sklearn pipelines and am using this post as a guide for my code:
https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj
I am trying to encode ...
5
votes
3
answers
636
views
R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?
See R's {drake}. It allows you to define a reproducible pipeline
...
- 620
2
votes
1
answer
647
views
How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?
As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and ...
- 135
6
votes
1
answer
3k
views
how to pass parameters over sklearn pipeline's stages?
I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API.
So I builded a Sklearn Pipeline for that:
<...
- 231
0
votes
1
answer
2k
views
Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn
My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels ...
- 103
6
votes
1
answer
4k
views
Difference between sklearn make_pipeline and imblearn make_pipeline
Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
- 161
1
vote
1
answer
366
views
How to deal with evaluations in a multi step ML pipeline
Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data.
Ok, ...
- 111
1
vote
0
answers
46
views
Using pipelines with a cross validation of several models in scikit-learn
Is there a simple way to cross-validate several models using sklearn pipelines?
- 175
1
vote
2
answers
2k
views
sklearn FeatureUnion vs ColumnTransformer
I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
- 227
4
votes
2
answers
525
views
Scikit-learn pipeline with scaling, dimensionality reduction, average prediction of multiple regression models, and grid search cross validation
I would like to use a sklearn pipeline doing this :
( - ) scale the data ( StandardScaler )
( - ) reduce dimensionality ( PCA )
( - ) make a prediction with GradientBoostingRegressor() and ...
6
votes
2
answers
1k
views
Model ensemble with Spark or Scikit Learn
I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators.
Here is a reproducible of what I would like my model to do with the Spark api
<...
- 499
7
votes
0
answers
3k
views
Tensorflow v1 Dataset API AttributeError with ndim
I'd like to make pipeline for optimizing Gpu and Cpu.
Dataset
It's about 10000 datapoint and 4 description variables for the regression problem.
...
- 71
0
votes
1
answer
975
views
sklearn predict: IndexingError: ('Too many indexers', 'occurred at index <name>')
The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to ...
- 101
1
vote
1
answer
71
views
Best Data Science & Machine Learning Conferences 2019? [closed]
What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
- 294
0
votes
1
answer
49
views
How does "sameModel" mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?
I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
- 11
1
vote
0
answers
358
views
Which design pattern is better for data pipelines: batches or one at a time?
I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes.
One ...
- 111