Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

I want to apply pd.cut as a transformer in a pipeline, like this: ...
0
votes
2answers
19 views

Error when trying to run RandomForestClassifer with Pipieline and GridSearch

I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV I am getting an error when I fit the data. I am not sure how to fix it. Will appreciate any help on this. My code is: ...
0
votes
0answers
9 views

Resources for building data pipelines

I am finding it difficult to find sources which walk you through building a data pipeline from scratch, for example from: - SQL Server (where I want to do my cleaning/ETL) - R/Python (where I want to ...
0
votes
0answers
11 views

Using sklearn's make_pipeline output doesn't match between test dataframe and output dataframe

I have a simple sklearn pipeline defined as below and I create a train_test split to fit and test my model. The R2-score looks ...
1
vote
1answer
19 views

How does imblearn apply the transformations during prediction?

Let's say I have a sklearn pipeline that: Imputes the data Randomly oversamples the minority class ...
0
votes
0answers
18 views

Tensorflow Extension on BigQuery as the ExampleGen

I like to build a pipeline on newly launched Google AI Pipelines, I've done the tutorial on their website and ready to try it out with my own project. I like to ingest the data directly from BigQuery, ...
0
votes
0answers
13 views

Handle OneHot Encoder in a pipeline with unseen data

so I have my data and split it in the beginning in test and train set. Then I apply following Pipelines on it: ...
0
votes
1answer
21 views

Google DataFlow

I'm trying to build a Google dataflow pipeline through one of the posts in Medium. https://levelup.gitconnected.com/scaling-scikit-learn-with-apache-beam-251eb6fcf75b However, it seems like I'm ...
1
vote
0answers
14 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
2
votes
1answer
98 views

sklearn - SimpleImputer in an empty Pipeline

When building a Pipeline I'm ending up at a scenario that can be simplified like this: FeatureUnion(NumericalPipeline(steps), CategoricalPipeline(steps)) Since this is one intermediary step in a ...
2
votes
0answers
30 views

PyTorch equivalent of tf.Data

I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
1
vote
1answer
25 views

Pipeline heterogeneous data

I'm facing an issue I don't know how to solve, but as I'm a beginner probably there is an easy solution I can't find. I'm playing with the titanic dataset and I want to work with pipelines (In order ...
0
votes
0answers
12 views

Which statistical method to use for feature selection between numerical inputs and categorical input?

I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
2
votes
1answer
44 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
1
vote
1answer
41 views

How to apply dataset balancing techniques whilst using Pipeline in Sklearn?

I am new to Machine Learning and trying to construct machine learning models that adhere to good practice and not susceptible to biases. I have decided to use Sklearn's ...
1
vote
1answer
128 views

GridSearchCV using Random Forest Reg Pipeline

...
1
vote
0answers
289 views

ValueError: Found input variables with inconsistent numbers of samples: [2, 44] [closed]

Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned" Then i create a ML model using the above as one of my features. ...
2
votes
0answers
18 views

Logic Check: Building a SKLearn Pipeline

I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set. Background: I have a ...
0
votes
0answers
15 views

Project structure ML time-series forecasting

I'm about to start with a ML learning time-series forecasting project which includes large-quantities of (2-million) time-series stored in JSON-files. The first step will involve feature engineering....
1
vote
2answers
53 views

Sklean pipeline order

Is there a correct order I should put data transformations into a pipeline using Sklearn? Currently I have these items in my pipeline; Feature selection, skew removal, scaling, outlier removal, ...
0
votes
1answer
36 views

order of features importance after make_column_transformer and pipeline

I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor. ...
4
votes
1answer
109 views

Data Science Pipelines vs Common CD/CL

What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build,...
0
votes
0answers
174 views

SHAP Kernel explainer for my pipeline model

I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer. The data contains different types of ...
2
votes
0answers
15 views

Tools/tutorials for compiling corpora for NLP experiments?

I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
0
votes
0answers
29 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
1
vote
1answer
133 views

scikit-learn pipeline strategy with both numeric and categorical columns

We are implementing an sklearn pipeline as follows (pseudo code): ...
1
vote
1answer
53 views

Is it possible to change pandas column data type within a sklearn pipeline?

Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables. Basically I need squeeze in a: <...
0
votes
0answers
28 views

Segment Data in SKlearn Pipeline

I have one dataset, and I would like to split this dataset into two segments - representing two different groups of people. I have created two separate binary classifiers for these two groups. What ...
0
votes
1answer
493 views

Error encoding categorical features using sklearn pipelines

I am new to sklearn pipelines and am using this post as a guide for my code: https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj I am trying to encode ...
5
votes
3answers
245 views

R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?

See R's {drake}. It allows you to define a reproducible pipeline ...
2
votes
1answer
388 views

How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and ...
4
votes
1answer
514 views

how to pass parameters over sklearn pipeline's stages?

I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API. So I builded a Sklearn Pipeline for that: <...
0
votes
1answer
477 views

Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels ...
1
vote
0answers
41 views

Scalable Data Science Pipeline

I was looking at this repository (https://github.com/jeffheaton/jh-kaggle-util) and was wondering if there's a better approach to creating a scalable machine learning pipeline for data science ...
2
votes
1answer
504 views

Difference between sklearn make_pipeline and imblearn make_pipeline

Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
1
vote
1answer
42 views

How to deal with evaluations in a multi step ML pipeline

Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data. Ok, ...
1
vote
0answers
12 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
1
vote
0answers
31 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
0
votes
2answers
700 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
2
votes
1answer
141 views

Combining scaling, dimensionality reduction, prediction using sklearn pipeline

I would like to use a sklearn pipeline doing this : ( - ) scale the data ( StandardScaler ) ( - ) reduce dimensionality ( PCA ) ( - ) make a prediction with GradientBoostingRegressor() and ...
4
votes
2answers
580 views

Model ensemble with Spark or Scikit Learn

I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators. Here is a reproducible of what I would like my model to do with the Spark api <...
0
votes
1answer
420 views

sklearn predict: IndexingError: ('Too many indexers', 'occurred at index <name>')

The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to ...
1
vote
1answer
54 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
0
votes
1answer
42 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
0answers
295 views

Which design pattern is better for data pipelines: batches or one at a time?

I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes. One ...
4
votes
2answers
450 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...