Questions tagged [pipelines]
A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.
36
questions with no upvoted or accepted answers
7
votes
0
answers
3k
views
Tensorflow v1 Dataset API AttributeError with ndim
I'd like to make pipeline for optimizing Gpu and Cpu.
Dataset
It's about 10000 datapoint and 4 description variables for the regression problem.
...
- 71
2
votes
0
answers
312
views
PyTorch equivalent of tf.Data
I've created a pipeline using tf.Data (or more accurately a mix of Pandas and then tf.Data). ...
- 400
1
vote
2
answers
98
views
unsupervised anomaly detection for univariate fast frequency time series data?
I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer
This time series ...
- 1,754
1
vote
0
answers
115
views
Data Pipeline Best Practices
I am looking more for concepts or pointed in the right direction for best practices for a project I am working on. I am currently playing around with Luigi for a data pipeline. I followed the tutorial ...
- 11
1
vote
0
answers
66
views
Data Lineage/Traceability in Pipelines
I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal.
Does anyone ...
- 11
1
vote
0
answers
42
views
Efficient way of adding new columns to datamart without reprocessing complete pipeline
I have a software engineering background and relatively new to data engineering. I am building out tables/datamarts in our datalake for data scientists and analysts to use. We use Airflow for ...
- 11
1
vote
0
answers
20
views
Which statistical method to use for feature selection between numerical inputs and categorical input?
I have a classification problem where my inputs are all numerical and continuous, my outputs are categorical labels [1,0,-1]. My own domain knowledge and ...
- 193
1
vote
0
answers
46
views
Using pipelines with a cross validation of several models in scikit-learn
Is there a simple way to cross-validate several models using sklearn pipelines?
- 175
1
vote
0
answers
358
views
Which design pattern is better for data pipelines: batches or one at a time?
I come from a software engineering background and have a firm knowledge of best design patterns in that world, but with data science I feel like I'm making elementary design pattern mistakes.
One ...
- 111
0
votes
1
answer
24
views
Data transformation pipeline error
When I'm making data transformation pipeline on a dataset I keep getting error as " all the input array dimensions except for concatenation axis must match exactly, but along dimension 0, the ...
- 1
0
votes
0
answers
23
views
Visualize Catboost and XGBoost training process + Cross Validation
I want to optimize Catboost and XGBoost models and visualize this process such that:
Use 3-fold cross-validation
Use my own pre-processing pipeline (Missing value imputation, over- or undersampling)
...
- 61
0
votes
0
answers
13
views
Read from database every time a new record goes through a Spark pipeline
I'm using Spark to filter and transform every new record that goes through a data stream, the problem is that for each new record I need to read a table from a Cassandra database in order to have the ...
0
votes
1
answer
128
views
What is the best\correct data split approach over time-series data to compare performance of forecasting future data among ML and DL regressors?
Let's say I have dataset contains a timestamp (non-standard timestamp column without datetime format) as a single feature and count as Label/target to predict ...
- 335
0
votes
0
answers
19
views
Sklearn pipelines, applying transformations to feature engineered columns in previous steps
I am working on a sklearn pipeline for my binary classification problem. The pipeline should perform typical things like downcasting data types, scaling numerical values, one-hot and ordinal encoding ...
- 23
0
votes
0
answers
30
views
0
votes
0
answers
32
views
Memory Error when loading a txt file for an ML model
I am trying to run the Python code below:
...
- 113
0
votes
0
answers
39
views
How do SciKit-Learn Pipelines pass Data between Steps?
I would like to create some Custom Transformers and incorporate them in a SciKit-Learn Pipeline.
I'd like to pass more than just a Dataframe between the transformer ...
- 557
0
votes
0
answers
42
views
0
votes
1
answer
57
views
How to make a pipeline for Videos Dataset for TensorFlow [Sequence Matters] & train Model Effectively with Low Memory System
I am working on a Deep Learning project and I am facing an issue with the size of the dataset. I want to make a pipeline for video dataset [Sequence Matters]. Because if I try the load the whole ...
- 21
0
votes
0
answers
569
views
Sklearn pipeline and custom transformers to remove specific value from columns
I'm trying to use sklearn pipelines and custom transformers to do outlier removal. What I want to do is identify outliers using an IQR-filter, set the outlier values to 'OUTLIER' (not NaN), and then ...
- 23
0
votes
0
answers
284
views
Data type conversion as part of sklearn pipeline
I'm currently learning and implementing sklearn pipelines. As an early step in the pipeline, I want to convert some data types that are wrong from the import (primarily object to numerical and object ...
- 23
0
votes
0
answers
7
views
Storing meta information from preprocessing step
Let's say I have a preprocessing step which removes some columns, replaces rare categories with the keyword "Other" and so on. All of that is done based on an initial data analysis.
Now, I ...
- 103
0
votes
0
answers
53
views
0
votes
0
answers
19
views
Most common transformations in an sklearn pipeline?
I'm wondering if there is a list anywhere of the most commonly used transformations that people use in a machine learning pipeline - particularly in Python and scikit-learn.
Additionally, are there ...
- 101
0
votes
1
answer
43
views
SkLearn DecisionTree doesn't include numerical variables after one hot encoding pipeline
I'm trying to fit a dataframe with SkLearn DecisionTree with the following code. But I get a error Length of feature_names, 9 does not match number of features, 8. ...
0
votes
1
answer
733
views
Error getting prediction explanation using shap_values when using scikit-learn pipeline?
I am building an NLP model to predict language type (C/C++/C#/Python...) for a given code.
Now I need to provide an explanation for my model prediction. For example the following user_input is written ...
- 133
0
votes
0
answers
111
views
How to retrain sklearn pipeline with new data?
I have trained and saved a data processing pipeline and an LGBM regressor on 3 months of historical data. Now I know that I can retrain the LGBM regressor on new data every day by passing my trained ...
0
votes
0
answers
147
views
Does sklearn.pipeline have a single mechanism for cross-validation regardless of model API?
With a single standard interface (sklearn.pipeline) on top of different regressors, how do I use cross-validation?
The example below uses two regressors with different internal cross-validation ...
- 127
0
votes
1
answer
2k
views
Shap after scikit pipeline - feature name
I'm using pipeline to transform data and predict model and I want to apply SHAP after that. However, when I apply it, it returns SHAP chart just fine, but the name of the feature are like feature 1, ...
- 31
0
votes
1
answer
25
views
determining size of batch, time of sending and memory in to send from scala to ML section
I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus.
...
- 1,754
0
votes
0
answers
407
views
Pipelines with categorical and nan values
I am trying a Regression model on a dataset which has categorical and numerical variables along with nan values. I want to use Pipelines for imputation and encoding purposes. Now I have a few ...
- 1,576
0
votes
0
answers
18
views
Create a pipeline for recursive eliminartion using artifical neural nets
I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated....
0
votes
0
answers
127
views
Apply two different Sklearn classifiers to two different subsets of the same data
I have a dataset that I need to run through a classification Pipeline. The dataset has 2 types of rows:
described: description column POPULATED
non-described: <...
0
votes
0
answers
2k
views
SHAP Kernel explainer for my pipeline model
I am trying to use SHAP kernel explainer to understand my XGBOOST model. My data is the lending club data and I am trying to predict the Grade of each customer.
The data contains different types of ...
- 53
0
votes
0
answers
35
views
Different extraction pipeline for train and test
I'm trying to create a production-ready ML model.
The problem is as follows:
Training data: Database A + Plus python aggregations.
Testing data: Database B + Plus python aggregations.
Both ...
- 101
0
votes
1
answer
975
views
sklearn predict: IndexingError: ('Too many indexers', 'occurred at index <name>')
The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to ...
- 101