Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
1
vote
1answer
18 views

cost-complexity-pruning-path with pipeline

I'm using Kaggle's titanic set. I'm using pieplines and I'm trying to prune my decision tree and for that I want the cost_complexity_pruning_path. The last line of code produces the error: ...
-1
votes
0answers
12 views

ML project - data pipeline framework - why always DAG?

I am looking to transform my project using some tested framework for data pipelines e.g. Kedro My question is: Is there a Ml framework that allows cycles in it's representations? So far it seems most ...
0
votes
1answer
28 views

How to remove features from a sklearn pipeline after it has already been fitted?

Background: I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to ...
1
vote
0answers
18 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
1
vote
0answers
32 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
0
votes
1answer
21 views

How is the target variable passed to the final estimator in this pipeline?

There is a pipeline like below. X is features and y is the target variable. I would like to know how ...
0
votes
0answers
16 views

Create a pipeline for recursive eliminartion using artifical neural nets

I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated....
0
votes
1answer
38 views

Compare cross validation and test set results

I am having a hard time understanding the results of a cross validation test and a test run on a test set. First I made the following pipeline: ...
1
vote
0answers
34 views

ML Pipeline on AWS SageMaker: How to create long-running query/preprocessing tasks

I'm a software engineer transitioning toward machine learning engineering, but need some assistance. I'm currently using AWS Lambda and Step Functions to run query and preprocessing jobs for my ML ...
-1
votes
1answer
123 views

Deep Learning Pipeline motivation

A Deep Learning Pipeline consists of the following 5 points: Define and prepare problem Summarize and understand data Process and prepare data Evaluate algorithms Improve results Here is the source: ...
0
votes
0answers
12 views

Time-series data pipeline for Kafka + Spark?

I'm working on a project related to predicting syscall sequences' malice. The data, which are syscall sequences that each belongs to a program, will be stored temporary to Kafka (which I have no idea ...
0
votes
2answers
151 views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
0
votes
0answers
33 views

Apply two different Sklearn classifiers to two different subsets of the same data

I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows: described: description column POPULATED non-described: <...
0
votes
2answers
22 views

Selecting Transforms with sklearn pipelines

So I am currently working on a Data set, and I want to use Pipelines to select the transforms. Here is an example of what I want to do : ...
6
votes
2answers
173 views

Possible harm in standardizing one-hot encoded features

While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)? Standardizing ...
0
votes
0answers
191 views

How to construct pipeline with different alternative transformations for different kind of features in Scikit-learn?

I try to construct a pipeline in sklearn where I do different (in some cases multiple) transformations on different kind (numeric/ordinal/binary nominal/non-binary non-ordinal nominal) features. An ...
2
votes
1answer
143 views

How to restrict the columns to be passed to final classifier in PMML Pipeline

I am working on building XGBoost PMML using SKLearn and SKLearn2PMML. I am having some numerical,somecategorical and datetime columns from which i am creating new feature inside the pipeline. When i ...
1
vote
1answer
63 views

Got some troubles with using OneHotEncoder to multiple categories

I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book). ...
1
vote
1answer
134 views

How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

I want to apply pd.cut as a transformer in a pipeline, like this: ...
0
votes
2answers
43 views

Error when trying to run RandomForestClassifer with Pipieline and GridSearch [closed]

I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV I am getting an error when I fit the data. I am not sure how to fix it. Will appreciate any help on this. My code is: ...
0
votes
0answers
10 views

Resources for building data pipelines

I am finding it difficult to find sources which walk you through building a data pipeline from scratch, for example from: - SQL Server (where I want to do my cleaning/ETL) - R/Python (where I want to ...
0
votes
0answers
19 views

Using sklearn's make_pipeline output doesn't match between test dataframe and output dataframe

I have a simple sklearn pipeline defined as below and I create a train_test split to fit and test my model. The R2-score looks ...
1
vote
1answer
49 views

How does imblearn apply the transformations during prediction?

Let's say I have a sklearn pipeline that: Imputes the data Randomly oversamples the minority class ...
0
votes
0answers
59 views

Tensorflow Extension on BigQuery as the ExampleGen

I like to build a pipeline on newly launched Google AI Pipelines, I've done the tutorial on their website and ready to try it out with my own project. I like to ingest the data directly from BigQuery, ...
0
votes
0answers
34 views

Handle OneHot Encoder in a pipeline with unseen data

so I have my data and split it in the beginning in test and train set. Then I apply following Pipelines on it: ...
0
votes
1answer
62 views

Google DataFlow

I'm trying to build a Google dataflow pipeline through one of the posts in Medium. https://levelup.gitconnected.com/scaling-scikit-learn-with-apache-beam-251eb6fcf75b However, it seems like I'm ...
1
vote
0answers
15 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
2
votes
1answer
493 views

sklearn - SimpleImputer in an empty Pipeline

When building a Pipeline I'm ending up at a scenario that can be simplified like this: FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps)) Since this is one intermediary step in a ...
2
votes
0answers
150 views

PyTorch equivalent of tf.Data

I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
1
vote
1answer
38 views

Pipeline heterogeneous data

I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find. I'm playing with the titanic dataset and I want to work with pipelines (In order ...
1
vote
0answers
17 views

Which statistical method to use for feature selection between numerical inputs and categorical input?

I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
2
votes
1answer
66 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
1
vote
1answer
485 views

How to apply dataset balancing techniques whilst using Pipeline in Sklearn?

I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's ...
1
vote
1answer
655 views

GridSearchCV using Random Forest Reg Pipeline

...
1
vote
0answers
802 views

ValueError: Found input variables with inconsistent numbers of samples: [2, 44] [closed]

Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned" Then i create a ML model using the above as one of my features. ...
2
votes
1answer
30 views

Logic Check: Building a SKLearn Pipeline

I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set. Background: I have a ...
1
vote
2answers
67 views

Sklean pipeline order

Is there a correct order I should put data transformations into a pipeline using Sklearn? Currently I have these items in my pipeline; Feature selection, skew removal, scaling, outlier removal, ...
0
votes
1answer
116 views

order of features importance after make_column_transformer and pipeline

I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor. ...
5
votes
1answer
247 views

Data Science Pipelines vs Common CD/CL

What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build,...
0
votes
0answers
446 views

SHAP Kernel explainer for my pipeline model

I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer. The data contains different types of ...
2
votes
0answers
17 views

Tools/tutorials for compiling corpora for NLP experiments? [closed]

I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
0
votes
0answers
29 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
1
vote
1answer
258 views

scikit-learn pipeline strategy with both numeric and categorical columns

We are implementing an sklearn pipeline as follows (pseudo code): ...
1
vote
1answer
100 views

Is it possible to change pandas column data type within a sklearn pipeline?

Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables. Basically I need squeeze in a: <...
0
votes
1answer
732 views

Error encoding categorical features using sklearn pipelines

I am new to sklearn pipelines and am using this post as a guide for my code: https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj I am trying to encode ...
5
votes
3answers
418 views

R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?

See R's {drake}. It allows you to define a reproducible pipeline ...
2
votes
1answer
512 views

How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and ...
4
votes
1answer
1k views

how to pass parameters over sklearn pipeline's stages?

I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API. So I builded a Sklearn Pipeline for that: <...
0
votes
1answer
744 views

Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels ...
1
vote
0answers
46 views

Scalable Data Science Pipeline

I was looking at this repository (https://github.com/jeffheaton/jh-kaggle-util) and was wondering if there's a better approach to creating a scalable machine learning pipeline for data science ...