Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
1
vote
1answer
19 views

GridSearchCV using Random Forest Reg Pipeline

...
1
vote
0answers
28 views

ValueError: Found input variables with inconsistent numbers of samples: [2, 44] [closed]

Have a piece of code where i am cleaning the text from the 'Description' column and storing it as "cleaned" Then i create a ML model using the above as one of my features. ...
2
votes
0answers
17 views

Logic Check: Building a SKLearn Pipeline

I am new to the concept of building a pipeline in SKLearn and would appreciate some sense-checking to ensure that I am not leaking info from my training sets into my test set. Background: I have a ...
0
votes
0answers
11 views

Project structure ML time-series forecasting

I'm about to start with a ML learning time-series forecasting project which includes large-quantities of (2-million) time-series stored in JSON-files. The first step will involve feature engineering....
1
vote
2answers
40 views

Sklean pipeline order

Is there a correct order I should put data transformations into a pipeline using Sklearn? Currently I have these items in my pipeline; Feature selection, skew removal, scaling, outlier removal, ...
0
votes
1answer
16 views

order of features importance after make_column_transformer and pipeline

I have a data preparation and model fitting pipeline that takes a dataframe (X_trn) and uses the ‘make_column_transformer’ and ‘Pipeline’ functions in sklearn to prepare the data and fit XGBRegressor. ...
4
votes
1answer
47 views

Data Science Pipelines vs Common CD/CL

What is the advantage of Data Science Specific CI/CD (kubeflow, Algo, TFX, mlflow, sagemaker pipelines) vs the already baked flavors that are more generic: Jenkins, Bamboo, Airflow, Google Cloud Build,...
0
votes
0answers
88 views

SHAP Kernel explainer for my pipeline model

I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer. The data contains different types of ...
1
vote
0answers
14 views

Tools/tutorials for compiling corpora for NLP experiments?

I have a couple of NLP ideas I want to try out (mostly for my own learning) - while I have the python/tensorflow background for running the actual training and prediction tasks, I don't have much ...
0
votes
0answers
29 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
1
vote
0answers
44 views

pipeline stragety when removing outliers

We are implementing an sklearn pipeline as follows (pseudo code): ...
1
vote
1answer
39 views

Is it possible to change pandas column data type within a sklearn pipeline?

Sklearn pipeline I am using has multiple transformers but one of the initial transformers returns numerical type and the consecutive one takes object type variables. Basically I need squeeze in a: <...
0
votes
0answers
26 views

Segment Data in SKlearn Pipeline

I have one dataset, and I would like to split this dataset into two segments - representing two different groups of people. I have created two separate binary classifiers for these two groups. What ...
0
votes
1answer
363 views

Error encoding categorical features using sklearn pipelines

I am new to sklearn pipelines and am using this post as a guide for my code: https://www.codementor.io/bruce3557/beautiful-machine-learning-pipeline-with-scikit-learn-uiqapbxuj I am trying to encode ...
5
votes
3answers
179 views

R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?

See R's {drake}. It allows you to define a reproducible pipeline ...
2
votes
1answer
333 views

How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and ...
2
votes
1answer
339 views

how to pass parameters over sklearn pipeline's stages?

I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API. So I builded a Sklearn Pipeline for that: <...
0
votes
1answer
291 views

Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

My input data is a (23948,) pandas.Series of strings containing newspaper headlines. My target are 20 labels of the headline (e.g. 'crime', 'politics') each binarily encoded with [0, 1]. The labels ...
1
vote
0answers
38 views

Scalable Data Science Pipeline

I was looking at this repository (https://github.com/jeffheaton/jh-kaggle-util) and was wondering if there's a better approach to creating a scalable machine learning pipeline for data science ...
2
votes
1answer
292 views

Difference between sklearn make_pipeline and imblearn make_pipeline

Can anybody please explain the difference between sklearn.pipeline.make_pipline and imblearn.pipeline.make_pipline.
1
vote
1answer
32 views

How to deal with evaluations in a multi step ML pipeline

Apache Spark provides us the pipeline to construct a ML pipeline. Moreover, we can evaluate the regressors in the pipeline, now here is the point I dont understand: how to apply to testing data. Ok, ...
1
vote
0answers
12 views

Preparing new samples for ML classifier with the same encoders

I was wondering what's the best way to preprocess new samples for my ML classifier. I have a raw data with about 3000 samples. I'm preprocessing it with some LabelEncoders and TargetEncdoders for ...
1
vote
0answers
31 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
0
votes
1answer
524 views

sklearn FeatureUnion vs ColumnTransformer

I am trying to build a sklearn pipeline which does different transformations on numerical data, and different transformation on categorical data. In the process, I compare the results from ...
0
votes
0answers
18 views

Data sharing between Custom Tranformer and estimator in pyspark

Using Titanic Dataset for classification. Created custom Estimator and transformer for pipeline. some of logical operation need to be done in fit() method and data ...
2
votes
1answer
108 views

Combining scaling, dimensionality reduction, prediction using sklearn pipeline

I would like to use a sklearn pipeline doing this : ( - ) scale the data ( StandardScaler ) ( - ) reduce dimensionality ( PCA ) ( - ) make a prediction with GradientBoostingRegressor() and ...
4
votes
2answers
505 views

Model ensemble with Spark or Scikit Learn

I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators. Here is a reproducible of what I would like my model to do with the Spark api <...
0
votes
1answer
293 views

sklearn predict: IndexingError: ('Too many indexers', 'occurred at index <name>')

The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to ...
1
vote
1answer
54 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
0
votes
1answer
42 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
0answers
259 views

Which design pattern is better for data pipelines: batches or one at a time?

I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes. One ...
4
votes
2answers
381 views

What is the meaning of the term “pipeline” within data science?

Note: this question was asked and removed just before I posted my answer below, so am repeating the general idea here People often refer to pipelines when talking about models, data and even layers ...