Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
0
votes
1answer
8 views

Retrieving the ordinal encoding of a variable after it's placed in a pipeline/columntransformer

I am applying ordinal encoding to a dataset through a column transformer - how can I retrieve the ordinal encoding of a feature (e.g. Area)? ...
0
votes
0answers
12 views

Does sklearn.pipeline have a single mechanism for cross-validation regardless of model API?

With a single standard interface (sklearn.pipeline) on top of different regressors, how do I use cross-validation? The example below uses two regressors with different internal cross-validation ...
0
votes
1answer
27 views

Custom vectorizer transformer in sklearn with cross validation

I created a custom transformer class called Vectorizer() that inherits from sklearn's ...
0
votes
1answer
13 views

Explaining the logic behind the pipe_line method for cross-validation of imbalance datasets

Reading the following article: https://kiwidamien.github.io/how-to-do-cross-validation-when-upsampling-data.html There is an explanation of how to use ...
0
votes
1answer
39 views

How to combine preprocessor/estimator selection with hyperparameter tuning using sklearn pipelines?

I'm aware of how to use sklearn.pipeline.Pipeline() for simple and slightly more complicated use cases alike. I know how to set up pipelines for homogeneous as well ...
0
votes
1answer
30 views

Custom preprocessing using piplines

I have searched a lot for this issue but unfortunately came up with nothing. Usually in a ML model, during preprocessing, we use Pipelines and ...
0
votes
1answer
48 views

ColumnTransformer worse performance than sklearn pipeline

I have an (unbalanced , binary data) pipeline model consisting of two pipelines (preprocessing and the actual model). Now I wanted to include SimpleImputer into my ...
1
vote
2answers
62 views

Deep Neural Network Model in sklearn Pipeline

Is it possible to add a deep neural network model as the estimator/model in an sklearn Pipeline? or is it only possible for ML models as the estimator. For example, ...
0
votes
0answers
85 views

Shap after scikit pipeline - feature name

I'm using pipeline to transform data and predict model and I want to apply SHAP after that. However, when I apply it, it returns SHAP chart just fine, but the name of the feature are like feature 1, ...
1
vote
1answer
54 views

How can I get the dataframe after scikit pipeline?

I'm making many data transformations and fitting a model using scikit pipeline, but I need to extract X_train and X_test right after transformations (imputer, encoding, etc) in order to use it for ...
1
vote
1answer
40 views

when I only give command 'fit', my class does 'transform' too

I have created 2 classes, first of which is: ...
4
votes
1answer
103 views

What can I do when my test and validation scores are good, but the submission is terrible?

This is a very broad question, I understand and I'm totally fine if someone believes it's not appropriate to do it. But it's killing me not to understand this... Here's the thing, I'm doing a machine ...
0
votes
2answers
794 views

AttributeError: 'numpy.ndarray' object has no attribute 'fit' [closed]

I am relatively new to ML and in the process of learning pipelines. I am creating a pipeline of custom transformers and get this error: AttributeError: 'numpy.ndarray' object has no attribute 'fit'. ...
0
votes
1answer
105 views

How to get column names after One Hot Encoding when using Pipelines?

I am using Pipeline and ColumnTransformer to preprocess the data. Basically I am using them to impute null values, scale the ...
1
vote
2answers
60 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
0
votes
1answer
20 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
0
votes
0answers
99 views

Pipelines with categorical and nan values

I am trying a Regression model on a dataset which has categorical and numerical variables along with nan values. I want to use Pipelines for imputation and encoding purposes. Now I have a few ...
1
vote
1answer
60 views

Alternative to EC2 for running ML batch training jobs on AWS

We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training. Most the the pipeline is on Lambda, but Lambda is known to ...
0
votes
0answers
13 views

Do sklearn Pipelines automatically split big datasets in chunks for the transform method?

Do sklearn Pipelines automatically split big datasets in chunks for the transform method? Each transformer in the pipe has a transform method. It seems as sklearn by default pushes all X_train into ...
1
vote
0answers
29 views

Data Pipeline Best Practices

I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi for a data pipeline. I followed the tutorial ...
1
vote
1answer
36 views

TPOT machine learning

I trained a regression TPOT algorithm on Google Colab, where the output of the TPOT process is some boiler plate Python code as shown below. ...
1
vote
0answers
21 views

Working with others: Tidy data vs Pretty data [closed]

When working with someone whose background and skill level in data work may not be strong, how do you best make the argument for tidy data over "pretty" data? There are notes of what I want ...
0
votes
0answers
38 views

How to improve SGD classifier performance?

I am having some issues with SGD to run with a sample of approx 5000 obs for a classification problem with imbalance. I am doing ...
0
votes
0answers
9 views

How to create a pipeline out of this kaggle notebook and automate it with Luigi?

I am new to ML and I was wondering if there's anyway to convert this notebook into an ML pipeline and automate it with a tool like luigi? https://www.kaggle.com/olivierg13/wine-ratings-analysis-w-...
0
votes
0answers
97 views

struggling with sklearn Pipeline, FeatureUnion for NLP

I'm working on Kaggle dataset trying to classify in the Tweet is a disaster or not. I have "text" feature that I will transform using TF-IDF but I also want to use "keyword" ...
1
vote
1answer
204 views

How to use SMOTE in Stacking in SKLearn?

I have a data set X,y and split them to train and test data. ...
1
vote
1answer
2k views

Why GridSearchCV returns nan?

I am using gridsearchcv to tune the parameters of my model and I also use pipeline and cross-validation. When I run the model to tune the parameter of XGBoost, it returns ...
0
votes
0answers
9 views

Automatic caching (validation) system for pipelines?

The vast majority of my DS projects begin with the creation of a simple pipeline to read or convert the original files/db filter, extract and clean some dataset which has as a result a dataset I ...
1
vote
1answer
78 views

cost-complexity-pruning-path with pipeline

I'm using Kaggle's titanic set. I'm using pieplines and I'm trying to prune my decision tree and for that I want the cost_complexity_pruning_path. The last line of code produces the error: ...
0
votes
1answer
407 views

How to remove features from a sklearn pipeline after it has already been fitted?

Background: I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to ...
1
vote
0answers
38 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
2
votes
2answers
216 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
0
votes
1answer
21 views

How is the target variable passed to the final estimator in this pipeline?

There is a pipeline like below. X is features and y is the target variable. I would like to know how ...
0
votes
0answers
16 views

Create a pipeline for recursive eliminartion using artifical neural nets

I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated....
0
votes
1answer
51 views

Compare cross validation and test set results

I am having a hard time understanding the results of a cross validation test and a test run on a test set. First I made the following pipeline: ...
-1
votes
1answer
133 views

Deep Learning Pipeline motivation

A Deep Learning Pipeline consists of the following 5 points: Define and prepare problem Summarize and understand data Process and prepare data Evaluate algorithms Improve results Here is the source: ...
1
vote
2answers
801 views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
0
votes
0answers
74 views

Apply two different Sklearn classifiers to two different subsets of the same data

I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows: described: description column POPULATED non-described: <...
0
votes
2answers
27 views

Selecting Transforms with sklearn pipelines

So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do : ...
7
votes
2answers
686 views

Possible harm in standardizing one-hot encoded features

While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)? Standardizing ...
2
votes
1answer
458 views

How to restrict the columns to be passed to final classifier in PMML Pipeline

I am working on building XGBoost PMML using SKLearn and SKLearn2PMML. I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i ...
1
vote
1answer
602 views

Got some troubles with using OneHotEncoder to multiple categories

I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book). ...
1
vote
1answer
246 views

How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

I want to apply pd.cut as a transformer in a pipeline, like this: ...
0
votes
2answers
127 views

Error when trying to run RandomForestClassifer with Pipieline and GridSearch [closed]

I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV I am getting an error when I fit the data. I am not sure how to fix it. Will appreciate any help on this. My code is: ...
1
vote
1answer
212 views

How does imblearn apply the transformations during prediction?

Let's say I have a sklearn pipeline that: Imputes the data Randomly oversamples the minority class ...
0
votes
1answer
144 views

Google DataFlow

I'm trying to build a Google dataflow pipeline through one of the posts in Medium. https://levelup.gitconnected.com/scaling-scikit-learn-with-apache-beam-251eb6fcf75b However, it seems like I'm ...
1
vote
0answers
15 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
2
votes
1answer
1k views

sklearn - SimpleImputer in an empty Pipeline

When building a Pipeline I'm ending up at a scenario that can be simplified like this: FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps)) Since this is one intermediary step in a ...
2
votes
0answers
248 views

PyTorch equivalent of tf.Data

I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
1
vote
1answer
64 views

Pipeline heterogeneous data

I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find. I'm playing with the titanic dataset and I want to work with pipelines (In order ...