Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [pipelines]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

31 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
0 answers
3k views

Tensorflow v1 Dataset API AttributeError with ndim

I'd like to make pipeline for optimizing Gpu and Cpu. Dataset It's about 10000 datapoint and 4 description variables for the regression problem. ...
AutomaKen's user avatar
2 votes
0 answers
331 views

PyTorch equivalent of tf.Data

I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
David Waterworth's user avatar
1 vote
0 answers
31 views

Integrating MLFlow and SageMaker for a More Robust ML Model Deployment Pipeline

I'm seeking advice on enhancing the deployment pipeline of a machine learning model that's accessed via a FastApi in production. My goal is to replace the existing setup with a more robust and ...
Daniel Ben Zaken's user avatar
1 vote
0 answers
103 views

How do SciKit-Learn Pipelines pass Data between Steps?

I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline. I'd like to pass more than just a Dataframe between the transformer ...
Connor's user avatar
  • 651
1 vote
2 answers
119 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
user10296606's user avatar
  • 1,844
1 vote
0 answers
144 views

What is the best practices of ETL pipeline using Luigi python package?

I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi package for the ETL pipeline. I followed the ...
ghawes's user avatar
  • 111
1 vote
0 answers
78 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
Gustav1985's user avatar
1 vote
0 answers
70 views

Efficient way of adding new columns to datamart without reprocessing complete pipeline

I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
Abdul R's user avatar
  • 11
1 vote
0 answers
22 views

Which statistical method to use for feature selection between numerical inputs and categorical input?

I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
Hamish Gibson's user avatar
1 vote
0 answers
58 views

Using pipelines with a cross validation of several models in scikit-learn

Is there a simple way to cross-validate several models using sklearn pipelines?
I.D.M's user avatar
  • 175
1 vote
0 answers
360 views

Which design pattern is better for data pipelines: batches or one at a time?

I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes. One ...
Breck's user avatar
  • 111
0 votes
0 answers
9 views

Feedback on this plan to make a data pipeline

I am picturing something like the following: List internet sources you want to acquire data from. Study the source code structure for each site. Try to build an “extractor” for each individual site ...
Julius Hamilton's user avatar
0 votes
0 answers
14 views

Data Transformation Pipeline Issue on Flight Fare Prediction End-to-End Machine Learning Project

I am following the process shown on Wine Quality Prediction End-to-End ML Project on Krish Naik's YouTube channel to do a Flight Fare Prediction Project. I am facing an issue with Data Transformation ...
Md. Ehsanul Haque Kanan's user avatar
0 votes
0 answers
20 views

How to handle multiple training jobs in AWS Sagemaker Pipelines?

I am trying to create a pipeline for training an image dataset via Sagemaker Pipelines. Based on the examples I understood that for all distinct stages like data preparation, model training, ...
Mimansa Maheshwari's user avatar
0 votes
0 answers
56 views

How does TensorFlow method tf.data.Dataset.reduce() work?

I was trying to compute the mean of the images that I fetched from a GCS bucket using TensorFlow's tf.data input pipeline. For this, I came across two methods: ...
Harsh Khare's user avatar
0 votes
0 answers
56 views

Create a TensorFlow Input pipeline from GCS Bucket?

I wish to generate a TensorFlow input pipeline (i.e. tf.data pipeline) for an image classification project. The images stored in a GCS bucket having access controls and is not publicly accessible. And ...
Harsh Khare's user avatar
0 votes
1 answer
75 views

Data transformation pipeline error

When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
Amy's user avatar
  • 1
0 votes
1 answer
449 views

What is the best\correct data split approach over time-series data to compare performance of forecasting future data among ML and DL regressors?

Let's say I have dataset contains a timestamp (non-standard timestamp column without datetime format) as a single feature and count as Label/target to predict ...
Mario's user avatar
  • 400
0 votes
0 answers
43 views

Memory Error when loading a txt file for an ML model

I am trying to run the Python code below: ...
lynx's user avatar
  • 113
0 votes
1 answer
126 views

How to make a pipeline for Videos Dataset for TensorFlow [Sequence Matters] & train Model Effectively with Low Memory System

I am working on a Deep Learning project and I am facing an issue with the size of the dataset. I want to make a pipeline for video dataset [Sequence Matters]. Because if I try the load the whole ...
Sharjeel M. Rajput's user avatar
0 votes
0 answers
1k views

Sklearn pipeline and custom transformers to remove specific value from columns

I'm trying to use sklearn pipelines and custom transformers to do outlier removal. What I want to do is identify outliers using an IQR-filter, set the outlier values to 'OUTLIER' (not NaN), and then ...
fendrbud's user avatar
0 votes
1 answer
1k views

Error getting prediction explanation using shap_values when using scikit-learn pipeline?

I am building an NLP model to predict language type (C/C++/C#/Python...) for a given code. Now I need to provide an explanation for my model prediction. For example the following user_input is written ...
SteveS's user avatar
  • 133
0 votes
0 answers
203 views

How to retrain sklearn pipeline with new data?

I have trained and saved a data processing pipeline and an LGBM regressor on 3 months of historical data. Now I know that I can retrain the LGBM regressor on new data every day by passing my trained ...
shobhit_kulshreshtha's user avatar
0 votes
0 answers
161 views

Does sklearn.pipeline have a single mechanism for cross-validation regardless of model API?

With a single standard interface (sklearn.pipeline) on top of different regressors, how do I use cross-validation? The example below uses two regressors with different internal cross-validation ...
user5406764's user avatar
0 votes
1 answer
28 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
user10296606's user avatar
  • 1,844
0 votes
0 answers
486 views

Pipelines with categorical and nan values

I am trying a Regression model on a dataset which has categorical and numerical variables along with nan values. I want to use Pipelines for imputation and encoding purposes. Now I have a few ...
spectre's user avatar
  • 2,105
0 votes
0 answers
18 views

Create a pipeline for recursive eliminartion using artifical neural nets

I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated....
Kanan H Patel's user avatar
0 votes
0 answers
151 views

Apply two different Sklearn classifiers to two different subsets of the same data

I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows: described: description column POPULATED non-described: <...
José Angel Rodríguez Ramos's user avatar
0 votes
0 answers
2k views

SHAP Kernel explainer for my pipeline model

I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer. The data contains different types of ...
Omar Baz's user avatar
0 votes
0 answers
36 views

Different extraction pipeline for train and test

I'm trying to create a production-ready ML model. The problem is as follows: Training data: Database A + Plus python aggregations. Testing data: Database B + Plus python aggregations. Both ...
Abhinav Ralhan's user avatar
0 votes
1 answer
1k views

sklearn predict: IndexingError: ('Too many indexers', 'occurred at index <name>')

The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to ...
Harvey's user avatar
  • 101