Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

Create a pipeline for recursive eliminartion using artifical neural nets

I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated....
0
votes
1answer
34 views

Compare cross validation and test set results

I am having a hard time understanding the results of a cross validation test and a test run on a test set. First I made the following pipeline: ...
0
votes
0answers
18 views

ML Pipeline on AWS SageMaker: How to create long-running query/preprocessing tasks

I'm a software engineer transitioning toward machine learning engineering, but need some assistance. I'm currently using AWS Lambda and Step Functions to run query and preprocessing jobs for my ML ...
-1
votes
1answer
122 views

Deep Learning Pipeline motivation

A Deep Learning Pipeline consists of the following 5 points: Define and prepare problem Summarize and understand data Process and prepare data Evaluate algorithms Improve results Here is the source: ...
0
votes
0answers
12 views

Time-series data pipeline for Kafka + Spark?

I'm working on a project related to predicting syscall sequences' malice. The data, which are syscall sequences that each belongs to a program, will be stored temporary to Kafka (which I have no idea ...
0
votes
2answers
87 views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
0
votes
0answers
24 views

Apply two different Sklearn classifiers to two different subsets of the same data

I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows: described: description column POPULATED non-described: <...
0
votes
2answers
22 views

Selecting Transforms with sklearn pipelines

So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do : ...
6
votes
2answers
105 views

Possible harm in standardizing one-hot encoded features

While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)? Standardizing ...
0
votes
0answers
102 views

How to construct pipeline with different alternative transformations for different kind of features in Scikit-learn?

I try to construct a pipeline in sklearn where I do different (in some cases multiple) transformations on different kind (numeric/ordinal/binary nominal/non-binary non-ordinal nominal) features. An ...
2
votes
1answer
52 views

How to restrict the columns to be passed to final classifier in PMML Pipeline

I am working on building XGBoost PMML using SKLearn and SKLearn2PMML. I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i ...
1
vote
1answer
23 views

Got some troubles with using OneHotEncoder to multiple categories

I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book). ...
1
vote
1answer
83 views

How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

I want to apply pd.cut as a transformer in a pipeline, like this: ...
0
votes
2answers
39 views

Error when trying to run RandomForestClassifer with Pipieline and GridSearch [closed]

I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV I am getting an error when I fit the data. I am not sure how to fix it. Will appreciate any help on this. My code is: ...
0
votes
0answers
10 views

Resources for building data pipelines

I am finding it difficult to find sources which walk you through building a data pipeline from scratch, for example from: - SQL Server (where I want to do my cleaning/ETL) - R/Python (where I want to ...
0
votes
0answers
14 views

Using sklearn's make_pipeline output doesn't match between test dataframe and output dataframe

I have a simple sklearn pipeline defined as below and I create a train_test split to fit and test my model. The R2-score looks ...
1
vote
1answer
29 views

How does imblearn apply the transformations during prediction?

Let's say I have a sklearn pipeline that: Imputes the data Randomly oversamples the minority class ...
0
votes
0answers
43 views

Tensorflow Extension on BigQuery as the ExampleGen

I like to build a pipeline on newly launched Google AI Pipelines, I've done the tutorial on their website and ready to try it out with my own project. I like to ingest the data directly from BigQuery, ...
0
votes
0answers
22 views

Handle OneHot Encoder in a pipeline with unseen data

so I have my data and split it in the beginning in test and train set. Then I apply following Pipelines on it: ...
0
votes
1answer
46 views

Google DataFlow

I'm trying to build a Google dataflow pipeline through one of the posts in Medium. https://levelup.gitconnected.com/scaling-scikit-learn-with-apache-beam-251eb6fcf75b However, it seems like I'm ...
1
vote
0answers
14 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
2
votes
1answer
359 views

sklearn - SimpleImputer in an empty Pipeline

When building a Pipeline I'm ending up at a scenario that can be simplified like this: FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps)) Since this is one intermediary step in a ...
2
votes
0answers
102 views

PyTorch equivalent of tf.Data

I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
1
vote
1answer
31 views

Pipeline heterogeneous data

I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find. I'm playing with the titanic dataset and I want to work with pipelines (In order ...
0
votes
0answers
15 views

Which statistical method to use for feature selection between numerical inputs and categorical input?

I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
2
votes
1answer
53 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
1
vote
1answer
297 views

How to apply dataset balancing techniques whilst using Pipeline in Sklearn?

I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's ...
1
vote
1answer
428 views

GridSearchCV using Random Forest Reg Pipeline

...
1
vote
0answers
709 views

ValueError: Found input variables with inconsistent numbers of samples: [2, 44] [closed]

Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned" Then i create a ML model using the above as one of my features. ...
2
votes
0answers
19 views

Logic Check: Building a SKLearn Pipeline

I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set. Background: I have a ...
0
votes
0answers
17 views

Project structure ML time-series forecasting

I'm about to start with a ML learning time-series forecasting project which includes large-quantities of (2-million) time-series stored in JSON-files. The first step will involve feature engineering....
1
vote
2answers
63 views

Sklean pipeline order

Is there a correct order I should put data transformations into a pipeline using Sklearn? Currently I have these items in my pipeline; Feature selection, skew removal, scaling, outlier removal, ...
0
votes
1answer
81 views

order of features importance after make_column_transformer and pipeline

I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor. ...
5
votes
1answer
192 views

Data Science Pipelines vs Common CD/CL

What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build,...
0
votes
0answers
358 views

SHAP Kernel explainer for my pipeline model

I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer. The data contains different types of ...
2
votes
0answers
17 views

Tools/tutorials for compiling corpora for NLP experiments? [closed]

I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
0
votes
0answers
29 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
1
vote
1answer
198 views

scikit-learn pipeline strategy with both numeric and categorical columns

We are implementing an sklearn pipeline as follows (pseudo code): ...
1
vote
1answer
83 views

Is it possible to change pandas column data type within a sklearn pipeline?

Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables. Basically I need squeeze in a: <...
0
votes
1answer
652 views

Error encoding categorical features using sklearn pipelines

I am new to sklearn pipelines and am using this post as a guide for my code: https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj I am trying to encode ...
5
votes
3answers
348 views

R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?

See R's {drake}. It allows you to define a reproducible pipeline ...
2
votes
1answer
461 views

How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and ...
4
votes
1answer
799 views

how to pass parameters over sklearn pipeline's stages?

I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API. So I builded a Sklearn Pipeline for that: <...
0
votes
1answer
638 views

Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels ...
1
vote
0answers
42 views

Scalable Data Science Pipeline

I was looking at this repository (https://github.com/jeffheaton/jh-kaggle-util) and was wondering if there's a better approach to creating a scalable machine learning pipeline for data science ...
5
votes
1answer
968 views

Difference between sklearn make_pipeline and imblearn make_pipeline

Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
1
vote
1answer
53 views

How to deal with evaluations in a multi step ML pipeline

Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data. Ok, ...
1
vote
0answers
33 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
0
votes
2answers
965 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
3
votes
1answer
163 views

Combining scaling, dimensionality reduction, prediction using sklearn pipeline

I would like to use a sklearn pipeline doing this : ( - ) scale the data ( StandardScaler ) ( - ) reduce dimensionality ( PCA ) ( - ) make a prediction with GradientBoostingRegressor() and ...