Questions tagged [preprocessing]
Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.
525
questions
0
votes
0
answers
31
views
Handling Missing Values
I have the following artificial data :
Now I am trying to handle missing values in the age and salary columns using mean imputation. I am using the following code to do so :
...
- 221
0
votes
0
answers
11
views
How to train model with data received from different 3x accelerometers sensors?
I want to make a model based on accelerometer data to recognise different activities like running, walking etc. I have a small dataset collected from my target sensor. I found another dataset with ...
- 101
0
votes
0
answers
8
views
Grouped Time Restricted Demand Regression with value cap
so I am working on quite an interesting regression task that I haven't encountered before.
Our company sells products (steel) in tons. We offer contracts where the customer orders a certain amount of ...
0
votes
0
answers
15
views
Image classification of centered objects with convolutional neural networks
Given that I have a set of images that contain multiple objects for which labels exist and the object the image label refers to is always in the center. The objects vary in size. I want to train a ...
- 1
1
vote
1
answer
54
views
What is the best way to avoid data leaking in timeseries forecasting multiple labels?
I'm predicting crypto pairs volume. I want to increase my accuracy by using one model for different pairs. Question is how to avoid timedata leaking?
Example n - pairs, t- time, m - features for each. ...
- 11
0
votes
1
answer
22
views
different range of target values in neural network
I am working on a neural network regression code. The dataset includes 14 features in the range value between -1 and 1. while the target variable is changing among (0.000759) to (1100).
The target ...
- 1
0
votes
2
answers
121
views
Recommended way to embed a text thousands of tokens long?
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
...
- 268
0
votes
1
answer
60
views
How to handle date columns in a dataset with over 100+ date columns?
Background: I have a dataset with around 85k rows and 320 columns.I have no formal domain knowledge and the columns ain't intuitive as well the dataset is not in a language that I speak or understand. ...
0
votes
3
answers
29
views
Data preprocessing
I just want to know how to determine whether to remove the missing values or to impute them with mean , median or mode. I usually remove the missing values but it decreases the size of the dataset by ...
0
votes
0
answers
7
views
TensorFlow lite classification model;{1,1,1,3}
How can I initialize and insert images into this classification model; what is meant by {1,1,1,3} according to my parameters it should be {1,size,size,3} please help
try {
Firstclass model = ...
0
votes
2
answers
37
views
Scaling of categorical feautres
In the context of algorithms that consider the scales of features, I have a situation where some features are encoded using ordinal encoding, some features are binary, and some features are standard ...
- 51
0
votes
1
answer
48
views
Clustering task: drop or not drop a categorical attribute/feature for which each row in the dataset contains a different value
I am dealing with a clustering task. In the dataset I am using there is a categorical feature and for each row in the dataset I have a different value for that feature (my dataset consists of 1000 ...
- 1
0
votes
0
answers
98
views
Can I convert a TensorFlow tensor to a CSV file?
I'm working on a semantic segmentation problem. I have saved my images in a tensor of shape (4767, 192, 192, 3) [It contains 4767 images of size 192192 with RGB channels]. On the other hand, I have ...
- 304
0
votes
0
answers
21
views
How to predict continous values using resnet regression model
I'm making a resnet regression model to predict force plate data using smartinsole data. so my data input is smartinsole data and the force plate is the label. below is an example of my data sample.
<...
- 31
1
vote
1
answer
48
views
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and no other ...
- 113
0
votes
0
answers
11
views
Transform monthly data to the weekly data based on categorical value
I have monthly data like this which is hours:
month,hours
1,20
2,19.5
3,21
...
I have weekly data like this which is quantities of the asset:
...
1
vote
0
answers
19
views
Labelling spectrograms
Currently I'm working on a ML project, just need an information, is there any tool that is present that can load audios file and generates spectrograms as well as an option to annotating/ label the ...
0
votes
0
answers
19
views
Problem with the size of test data?
I am working on my thesis, which has two different research questions:
Evaluate transformer models while incorporating non textuel features
Evaluate the importance of data quality in tranformer ...
- 1
0
votes
0
answers
15
views
Preparing LSTM Network input from multiple files
I would like to train LSTM Network, which should take 5 files as input and predict the 6th file. Each file contains 810000 data points (precipitation values), and each data point indices the location. ...
2
votes
1
answer
70
views
How to improve the preservation of the global data structure in UMAP?
I have a dataset, where the features are comprised of points arranged in a regular grid on a simplex. Each of these points are defined as follows: A point $\mathbf{x}$ on the simplex can be ...
- 121
0
votes
1
answer
51
views
Keras MLP not working
For my degree's final project I am working with Keras and trying to build different AI models. I'm having trouble with an MLP. First I preprocess the UNSW-NB15 dataset and later use it as input in a ...
0
votes
0
answers
11
views
Data processing - how to input a pandas column that contains numbers and numpy arrays
I have a pandas df that contains numbers and strings. I use word2vec to convert all the strings into embeddings. The problem now is that these embeddings are all numpy arrays. So now my pandas df ...
- 1
0
votes
0
answers
25
views
Why is the shape of my transformed data from CountVectorizer 2x2?
I am currently working on a text classification problem using Python and scikit-learn. I am using CountVectorizer to convert my text data into numerical vectors. I have also applied a preprocessor to ...
0
votes
0
answers
15
views
Performing imputation only on the test set?
I'm working on a medical machine learning problem. The key challenge is working with small datasets with quite a lot of missing data.
Experimentally, I've seen complete-case analysis (i.e. dropping ...
0
votes
0
answers
16
views
Best way to regrid data (polar stereographic to regular)
I have data that's in 'polar stereographic' form and I want to regrid it to a regular grid that matches a grid I currently have. I've seen a few example that tend to be fairly complicated. I'd ideally ...
- 91
1
vote
0
answers
55
views
How to encode & scale IP addresses as input for ML models
Im currently working on an anomaly detection while making a transaction. As a part of the data that I extracted, I have the IP addresses of the indivduals who made the transaction. Since the IP ...
0
votes
2
answers
63
views
Fine-tune GPT on sketch data (stroke-3)
These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it.
I was planning on using some pretrained models that are ...
- 31
1
vote
1
answer
38
views
Order of preproccesing, avoiding leakage and metrics
I have a dataset with ~40k records and 16 columns (including the target) and I want to understand the correct process behind whole data science proccess.
This is what I did:
Performed an EDA which ...
- 11
0
votes
0
answers
15
views
Why not decrease the number of filters in each 1-dimensional convolutional layer for spectrogram processing?
In convolutional neural networks, the number of filters usually increases with every convolutional layer. Why is this common practice?
Nonetheless, these networks typically process images. When ...
0
votes
1
answer
33
views
Is there a standard order of operations for data preparation?
In what order should I do the following given a dataset:
(E)ncoding of Categorical Variables
(N)ormalization
(B)alancing of data
(I)mputation of Missing Values
(R)emoval of Duplicates/Infinity/...
- 15
4
votes
1
answer
262
views
How can someone build a dataset for a "propensity to purchase" model?
Ok, this might seem a trivial question for some and it's not even a question, more like a discussion. I read the rules and I believe it's everything fine, so I'm gonna take my chances...
Here's the ...
- 41
1
vote
0
answers
47
views
Select rows in Orange based on column value comparison
I'm trying to remove some rows from a cardio dataset where the value for the systolic pressure (ap_hi) is below the diastolic pressure (ap_low) which to me indicates a data error. The code I would ...
0
votes
0
answers
15
views
What is the best way to whittle down rendundant categorical data?
I'm trying to build a linear regression model in Tensorflow (and preprocessing with pandas) that will help me categorize bank transactions. I'm trying to whittle down the vendor parameter, because the ...
0
votes
1
answer
29
views
How can I know the best number of features to use?
I noticed that developing ml models a very important step in feature engineering is adding new features that can explain better the target variable.
Recently I experienced a situation where by adding ...
0
votes
0
answers
10
views
What could be the most optimised way to run data pipeline which does calculations parallel
Our current implementation is like an api call and that creates different background threads to execute functions.
- 101
0
votes
1
answer
35
views
How to treat categorical columns after ordinal encoding?
If encode three categorical variables like "bad", "normal", "good" into 0,1,2, after that can I treat them as numerical values? So can I perform on them MinMaxScaler or ...
0
votes
0
answers
73
views
Best string similarity metric not considering word order
I'm sorry if the title is misleading, but I didn't really know how to explain what I am searching for.
I have a dataset containing two columns representing names and surnames of a bunch of people. ...
- 1
0
votes
0
answers
19
views
Can this task for phrases be called lemmatization?
I want to 'lemmatize' phrases to dictionary entries. For instance, the following collocates can be standardized to the idiom in the aforementioned link ...
- 496
0
votes
0
answers
14
views
Can I create my validation set after preprocessing the train set?
I have another question regarding the dataset split.
If I have a train and test set can I perform all the preprocessing steps (scaling, imputation, ...) on the training one and then split it into ...
0
votes
2
answers
143
views
When is the right moment to split the dataset?
I would like to ask a question about the dataset splitting.
If I have a dataset can I perform preprocessing (imputation, scaling, ecc.) on the entire dataset and then splitting it into train and ...
0
votes
0
answers
23
views
Sklearn pipelines, applying transformations to feature engineered columns in previous steps
I am working on a sklearn pipeline for my binary classification problem. The pipeline should perform typical things like downcasting data types, scaling numerical values, one-hot and ordinal encoding ...
- 23
0
votes
0
answers
225
views
keras model.fit() with data generator error
I want to use a DataGenerator but I get this error.
...
- 1
1
vote
1
answer
131
views
Preprocessing advice for large text corpus in natural language generation (NLG)
I have a large text corpus (i.e. 30 million sentences, all in lowercase in the format of Penn Treebank) that I want to use to train a neural network for natural language generation. What preprocessing ...
0
votes
0
answers
21
views
Data augmentation layer based on physical model for time series data
I am quite new to the Keras API, so forgive me if I use incorrect terminology and for my lack of knowledge about the API.
This is for a mathematical (wave modelling) research project and I am quite ...
- 101
0
votes
1
answer
89
views
Preprocessing vs Data Engineering vs Feature Engineering
I'm having a very hard categorizing different methods of data preparation into the 3 categories Preprocessing, Data Engineering, Feature Engineering.
A somewhat common definition describes Feature ...
0
votes
0
answers
31
views
How do I prepare data for a multivariate LSTM model that includes multiple patients
I want to predict the blood glucose levels using time series data with multiple features such as time, glucose levels, carbohydrates, fat, and protein. I have a dataset with hundreds of patients but ...
- 1
0
votes
2
answers
127
views
Should i remove french special characters and apostrophes
I am working on a french text preprocessing task, in order to prepare the data to train an NLP model. But I do not know if it is better to remove french special characters and apostrophes or keep them....
- 3
1
vote
1
answer
102
views
What is the right processing order when working with a dataset that already consists of test and train data?
I want to work on the following task:
Text Classification using Deep Learning models and a Transfer Learning model.
The notebook that I'm creating should include the following steps:
Data ...
- 135
0
votes
1
answer
63
views
What are more advanced categorical encoding methods?
I'm familiar with the common methods:
Label Encoding: {A, B, C} -> [0, 1, 2]
One-Hot Encoding: ...
- 11
3
votes
1
answer
643
views
Out of range [0,1] MinMaxScaler for test data
I know that for MinMaxScaler we should apply it to train data, then apply it, with the obtained parameters, over test data:
...
- 31