Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Feedback on this plan to make a data pipeline

I am picturing something like the following: List internet sources you want to acquire data from. Study the source code structure for each site. Try to build an “extractor” for each individual site ...
Julius Hamilton's user avatar
1 vote
0 answers
31 views

Integrating MLFlow and SageMaker for a More Robust ML Model Deployment Pipeline

I'm seeking advice on enhancing the deployment pipeline of a machine learning model that's accessed via a FastApi in production. My goal is to replace the existing setup with a more robust and ...
Daniel Ben Zaken's user avatar
0 votes
1 answer
37 views

Add tuning stage to DVC pipeline

I have an ML pipeline built with DVC that I use for experiment tracking. This allows running and tracking several experiments. Also, using hydra integration I can grid search hyper parameters. However,...
giulatona's user avatar
0 votes
0 answers
14 views

Data Transformation Pipeline Issue on Flight Fare Prediction End-to-End Machine Learning Project

I am following the process shown on Wine Quality Prediction End-to-End ML Project on Krish Naik's YouTube channel to do a Flight Fare Prediction Project. I am facing an issue with Data Transformation ...
Md. Ehsanul Haque Kanan's user avatar
0 votes
0 answers
20 views

How to handle multiple training jobs in AWS Sagemaker Pipelines?

I am trying to create a pipeline for training an image dataset via Sagemaker Pipelines. Based on the examples I understood that for all distinct stages like data preparation, model training, ...
Mimansa Maheshwari's user avatar
0 votes
0 answers
56 views

How does TensorFlow method tf.data.Dataset.reduce() work?

I was trying to compute the mean of the images that I fetched from a GCS bucket using TensorFlow's tf.data input pipeline. For this, I came across two methods: ...
Harsh Khare's user avatar
0 votes
0 answers
56 views

Create a TensorFlow Input pipeline from GCS Bucket?

I wish to generate a TensorFlow input pipeline (i.e. tf.data pipeline) for an image classification project. The images stored in a GCS bucket having access controls and is not publicly accessible. And ...
Harsh Khare's user avatar
0 votes
1 answer
74 views

Data transformation pipeline error

When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
Amy's user avatar
  • 1
0 votes
1 answer
440 views

What is the best\correct data split approach over time-series data to compare performance of forecasting future data among ML and DL regressors?

Let's say I have dataset contains a timestamp (non-standard timestamp column without datetime format) as a single feature and count as Label/target to predict ...
Mario's user avatar
  • 400
0 votes
0 answers
43 views

Memory Error when loading a txt file for an ML model

I am trying to run the Python code below: ...
lynx's user avatar
  • 113
0 votes
1 answer
268 views

Creating new features as linear combination of others as part of a scikit-learn pipeline?

I have a number of raw features that go into a scikit-learn model. I've already got a number of preprocessing steps (such as PolynomialFeatures) that creates additional features as part of my pipeline....
gammapoint's user avatar
0 votes
2 answers
45 views

Optimization of the entire model development process

I want to perform a global optimization of the entire model development pipeline. I have several stages of development, each of which can be performed automatically: preprocessing, removal of outliers/...
Andrew's user avatar
  • 406
1 vote
0 answers
102 views

How do SciKit-Learn Pipelines pass Data between Steps?

I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline. I'd like to pass more than just a Dataframe between the transformer ...
Connor's user avatar
  • 651
0 votes
2 answers
1k views

Does Scikit-Learn's OneHotEncoder make all Columns Categorical?

I've been using Scikit-Learn's OneHotEncoder to turn categorical data into binary columns, however, it seems that fitting ...
Connor's user avatar
  • 651
1 vote
1 answer
21 views

How to solve mismatched code dependency issues between two or more different ML models/structures/frameworks?

I'm fairly new to machine learning. Currently I am trying to build a pipeline that uses two established NLP models. One is BERT that is fairly easy to load and quite structured. Another is FLAIR which ...
Rafsan Ahmed's user avatar
0 votes
1 answer
125 views

How to make a pipeline for Videos Dataset for TensorFlow [Sequence Matters] & train Model Effectively with Low Memory System

I am working on a Deep Learning project and I am facing an issue with the size of the dataset. I want to make a pipeline for video dataset [Sequence Matters]. Because if I try the load the whole ...
Sharjeel M. Rajput's user avatar
1 vote
1 answer
641 views

Combining sklearn pipelines with different output shape

As part of a data preprocessing step, I'm trying to create a "master pipeline" from two separate pipelines, one for numerical features and one for datetime features. The numerical pipeline ...
fendrbud's user avatar
0 votes
0 answers
1k views

Sklearn pipeline and custom transformers to remove specific value from columns

I'm trying to use sklearn pipelines and custom transformers to do outlier removal. What I want to do is identify outliers using an IQR-filter, set the outlier values to 'OUTLIER' (not NaN), and then ...
fendrbud's user avatar
0 votes
1 answer
62 views

SkLearn DecisionTree doesn't include numerical variables after one hot encoding pipeline

I'm trying to fit a dataframe with SkLearn DecisionTree with the following code. But I get a error Length of feature_names, 9 does not match number of features, 8. ...
esokumamon's user avatar
0 votes
1 answer
1k views

Error getting prediction explanation using shap_values when using scikit-learn pipeline?

I am building an NLP model to predict language type (C/C++/C#/Python...) for a given code. Now I need to provide an explanation for my model prediction. For example the following user_input is written ...
SteveS's user avatar
  • 133
0 votes
1 answer
637 views

Need an example of a custom class whose instance is fed to sklearn Pipeline / make_pipeline to use with GridSearchCV

According to sklearn.pipeline.Pipeline documentation, the class whose instance is a pipeline element should implement fit() and transform(). I managed to create a custom class that has these methods ...
James's user avatar
  • 133
2 votes
2 answers
63 views

I cannot work out the benefits of a pipeline over a linear sequence of code instructions

I've looked at quite a number of how to 'create a pipeline' instructions, but I have yet to see an explanation of the benefits over what I am showing below. To keep my example code agnostic I'll use ...
ThomasAJ's user avatar
  • 121
0 votes
2 answers
621 views

Dynamic creation of sklearn pipeline

I am trying to create an automatic pipeline builder functionality that takes into account a large set of conditions such as the existence of missing values, the scale of numerical features, etc., and ...
lazarea's user avatar
  • 299
0 votes
0 answers
202 views

How to retrain sklearn pipeline with new data?

I have trained and saved a data processing pipeline and an LGBM regressor on 3 months of historical data. Now I know that I can retrain the LGBM regressor on new data every day by passing my trained ...
shobhit_kulshreshtha's user avatar
0 votes
1 answer
941 views

Retrieving the ordinal encoding of a variable after it's placed in a pipeline/columntransformer

I am applying ordinal encoding to a dataset through a column transformer - how can I retrieve the ordinal encoding of a feature (e.g. Area)? ...
lostwanderer's user avatar
0 votes
0 answers
161 views

Does sklearn.pipeline have a single mechanism for cross-validation regardless of model API?

With a single standard interface (sklearn.pipeline) on top of different regressors, how do I use cross-validation? The example below uses two regressors with different internal cross-validation ...
user5406764's user avatar
0 votes
1 answer
597 views

Custom vectorizer transformer in sklearn with cross validation

I created a custom transformer class called Vectorizer() that inherits from sklearn's ...
lazarea's user avatar
  • 299
0 votes
1 answer
218 views

Explaining the logic behind the pipe_line method for cross-validation of imbalance datasets

Reading the following article: https://kiwidamien.github.io/how-to-do-cross-validation-when-upsampling-data.html There is an explanation of how to use ...
PwNzDust's user avatar
  • 149
1 vote
1 answer
258 views

How to combine preprocessor/estimator selection with hyperparameter tuning using sklearn pipelines?

I'm aware of how to use sklearn.pipeline.Pipeline() for simple and slightly more complicated use cases alike. I know how to set up pipelines for homogeneous as well ...
lazarea's user avatar
  • 299
0 votes
1 answer
160 views

Custom preprocessing using piplines

I have searched a lot for this issue but unfortunately came up with nothing. Usually in a ML model, during preprocessing, we use Pipelines and ...
spectre's user avatar
  • 2,105
1 vote
1 answer
564 views

ColumnTransformer worse performance than sklearn pipeline

I have an (unbalanced , binary data) pipeline model consisting of two pipelines (preprocessing and the actual model). Now I wanted to include SimpleImputer into my ...
corianne1234's user avatar
1 vote
2 answers
649 views

Deep Neural Network Model in sklearn Pipeline

Is it possible to add a deep neural network model as the estimator/model in an sklearn Pipeline? or is it only possible for ML models as the estimator. For example, ...
DataPlug's user avatar
  • 258
0 votes
1 answer
3k views

Shap after scikit pipeline - feature name

I'm using pipeline to transform data and predict model and I want to apply SHAP after that. However, when I apply it, it returns SHAP chart just fine, but the name of the feature are like feature 1, ...
rebar's user avatar
  • 31
2 votes
1 answer
4k views

How can I get the dataframe after scikit pipeline?

I'm making many data transformations and fitting a model using scikit pipeline, but I need to extract X_train and X_test right after transformations (imputer, encoding, etc) in order to use it for ...
rebar's user avatar
  • 31
1 vote
1 answer
47 views

when I only give command 'fit', my class does 'transform' too

I have created 2 classes, first of which is: ...
greeksalad's user avatar
4 votes
1 answer
222 views

What can I do when my test and validation scores are good, but the submission is terrible?

This is a very broad question, I understand and I'm totally fine if someone believes it's not appropriate to do it. But it's killing me not to understand this... Here's the thing, I'm doing a machine ...
Yuxxxxxx's user avatar
  • 141
0 votes
2 answers
8k views

AttributeError: 'numpy.ndarray' object has no attribute 'fit' [closed]

I am relatively new to ML and in the process of learning pipelines. I am creating a pipeline of custom transformers and get this error: AttributeError: 'numpy.ndarray' object has no attribute 'fit'. ...
nagesh hatti's user avatar
1 vote
1 answer
3k views

How to get column names after One Hot Encoding when using Pipelines?

I am using Pipeline and ColumnTransformer to preprocess the data. Basically I am using them to impute null values, scale the ...
spectre's user avatar
  • 2,105
1 vote
2 answers
119 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
user10296606's user avatar
  • 1,844
0 votes
1 answer
28 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
user10296606's user avatar
  • 1,844
0 votes
0 answers
486 views

Pipelines with categorical and nan values

I am trying a Regression model on a dataset which has categorical and numerical variables along with nan values. I want to use Pipelines for imputation and encoding purposes. Now I have a few ...
spectre's user avatar
  • 2,105
1 vote
1 answer
161 views

Alternative to EC2 for running ML batch training jobs on AWS

We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training. Most the the pipeline is on Lambda, but Lambda is known to ...
Cybernetic's user avatar
1 vote
0 answers
143 views

What is the best practices of ETL pipeline using Luigi python package?

I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi package for the ETL pipeline. I followed the ...
ghawes's user avatar
  • 111
2 votes
1 answer
491 views

TPOT machine learning

I trained a regression TPOT algorithm on Google Colab, where the output of the TPOT process is some boiler plate Python code as shown below. ...
bbartling's user avatar
  • 403
1 vote
0 answers
25 views

Working with others: Tidy data vs Pretty data [closed]

When working with someone whose background and skill level in data work may not be strong, how do you best make the argument for tidy data over "pretty" data? There are notes of what I want ...
Rob Creel's user avatar
  • 111
1 vote
1 answer
743 views

How to use SMOTE in Stacking in SKLearn?

I have a data set X,y and split them to train and test data. ...
Aaron's user avatar
  • 201
4 votes
1 answer
10k views

Why GridSearchCV returns nan?

I am using gridsearchcv to tune the parameters of my model and I also use pipeline and cross-validation. When I run the model to tune the parameter of XGBoost, it returns ...
Aaron's user avatar
  • 201
1 vote
1 answer
285 views

cost-complexity-pruning-path with pipeline

I'm using Kaggle's titanic set. I'm using pieplines and I'm trying to prune my decision tree and for that I want the cost_complexity_pruning_path. The last line of code produces the error: ...
user5744148's user avatar
0 votes
1 answer
2k views

How to remove features from a sklearn pipeline after it has already been fitted?

Background: I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to ...
DataScienceRick's user avatar
1 vote
0 answers
78 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
Gustav1985's user avatar