Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
1answer
10 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
1
vote
1answer
49 views

How do you choose the min and max for the min-max normalization on a histogram classifier?

Please let me know what to do when there is a value in the testing set is bigger than the max value used to min-max normalize the training set building a histogram classifier. Do I go back and change ...
0
votes
1answer
339 views

LSTM - How to prepare train from a dataset which contains multiple observations for different events

I m using LSTM in a project related to MobiFall dataset which contains falls and daily activitives - such as walking, sitting etc - sensed by accelerometer, gyroscope and orientation sensors in x,y,z ...
0
votes
0answers
6 views

Small object detection preprocessing

I am currently working on an object detection project in which I amtrying to detect very small objects 50x50 object in a 2k image. EfficientDet produced a very low result if I just put the raw ...
1
vote
2answers
30 views

How to convert longitude and latitude in time series data from daily to weekly?

I have time series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
0
votes
0answers
14 views

How to fill missing latitude and longitude values in time series data?

I have time-series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
2
votes
1answer
693 views

Decision Tree - Preprocessing for very sparse features

How do we pre-process data for very sparse features for a decision tree? From this Turi documentation for decision trees It mentioned this: Why chose decision trees? Different kinds of models ...
0
votes
0answers
26 views

Predict on new data / Model Deployment

I do not undestand, how the deployment of a ML-model works in the reality. A given dataset needs to be mostly time pre-processed (for example One Hot Encoding). After will be a model cretead and ...
2
votes
2answers
695 views

normalization/denormalization for linear regression problem

My question is simple actually, I have two features that have big difference in scale. So I used a simple normalization by dividing the scale=np.max(array) for both data and lables. Then after ...
0
votes
1answer
13 views

Storing and loading bottleneck features for transfer learning on large data sets (Keras)

I would like to apply transfer learning on a pretty large image data set in order to solve a classification problem. Currently I load a pre-trained net without the top layers, add my own top layers, ...
0
votes
1answer
38 views

Can someone clarify what the linear assumption of PCA is?

0 For the past few hours I've been trying to search what this linear assumption is. Some of the articles states that that your independent variables have to be linear in relationship and need some ...
0
votes
1answer
10 views

Treat similar observations in a classification problem

I have a dataset of about 200k rows and I'm working on a classification problem. Grouping the dataset by a key variable, I noticed that some rows, with the same key value, have similar values in other ...
2
votes
2answers
59 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
0
votes
0answers
6 views

How to perform cropping on large file dataset using ImageGenerators?

I have a large dataset of RGB images with 220x220x3. I want to crop the region of interest from each of them. For that, I have defined my own function. Im trying to use this with DataImageGenerators ...
0
votes
1answer
4k views

ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

I want to create a new variable for the dataframe details called lower after iterating through multiple ...
0
votes
2answers
23 views

Multiple Features with the same categorical value

In working with a telecommunications data set with multiple categorical variables which all depend on Internet Service, a separate categorical variable, I ran into the problem where 'No Internet ...
30
votes
3answers
16k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
1
vote
1answer
104 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
0
votes
0answers
26 views

How to remove spikes from data with Python using signal.find_peaks

I need to make a regression model to estimate data values in future. Train set contains occasional spikes that make my model less accurate, thus I'm trying to locate and remove them. I've used ...
1
vote
1answer
409 views

Reconstituting estimated/predicted values to original scale from MinMaxScaler

Thank you all, I am playing around with a deterministic function in order to understand machine learning as in the tutorial blog at https://machinelearningmastery.com/tutorial-first-neural-network-...
1
vote
1answer
58 views

What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
0
votes
1answer
17 views

Extracting and tagging/labeling text from pdfs [closed]

I'm trying to extract paragraphs from 230 pdfs that have similar formats, and then label the paragraphs based on the pdf's section headings. Normally, I'd convert the pdf to txt files or extract the ...
1
vote
0answers
20 views

Dealing with null values in categorical string data

Which of the following is a better approach in dealing with null values and why? Fill the null values with the mode (most occurring value). Consider the null itself as a category by converting it to ...
1
vote
1answer
24 views

Log scaling whole dataset

I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value. I am wondering how I can get rid of this -infinity? I have been advised to add a small value ...
1
vote
2answers
18 views

How to properly use oversampling without inflating results?

I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i ...
0
votes
1answer
27 views

Logistics Demand Forecasting with 20k Different Time Series

I'm trying to tackle a very challenging problem and I would appreciate your help. My organization has a lot of different items which can be demanded by our clients. Those items can also be returned ...
0
votes
1answer
2k views

what is correct way to perform normalization on data in Auto encoder?

working on anomaly detection problem. i'm using auto-encoder to denoise given input. I trained network with normal data(anomaly free). so model predict normal state of given input. Normalization of ...
3
votes
2answers
112 views

Preprocessing: StandardScaler() Do we really need mean to be zero?

For instance, many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the l1 and l2 regularizers of linear models) assume that all ...
1
vote
0answers
18 views

Pre-processing packet data [closed]

I am confused about how to pre-processed packet data. Does anybody have an idea of how to encode network ports and IP addresses? My goal is to model the data in a such way that I will get a good ...
2
votes
1answer
388 views

Positively skewed target label in regression

I have a dataset where the target label is positively skewed and produces a long tail, and currently I have a high residual on these values when experimenting with some linear, tree-based and neural-...
0
votes
1answer
58 views

How to deal with name strings in large data sets for ML?

My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later. Some word embedding techniques were used for longer text ...
2
votes
1answer
29 views

Is feature scaling at all needed for a feature set with a single feature?

I understand that feature scaling is required to bring features in different magnitudes on a common scale so the model is not biased towards features with higher magnitudes. But if there is only a ...
0
votes
3answers
2k views

Discarding non-english words in column

I have some non-english words/sentences in my data. I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the ...
0
votes
2answers
136 views

Standardization with positive and negatives

I have a data set that has a few columns such as: Total cost: mean = 3,000,000 Percent complete: mean = 50 final profit %: mean = 14 I know with such different orders of magnitude before I fit a ...
1
vote
1answer
53 views

Data scaling for large dynamic range in neural networks

The usual strategy in neural networks today is to use min-max scaling to scale the input feature vector from 0 to 1. I want to know if the same principle holds true if our inputs have a large dynamic ...
12
votes
2answers
16k views

One Hot Encoding vs Word Embedding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
1
vote
1answer
128 views

Customize loss function for Music Generation LSTM (?)

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract ...
0
votes
1answer
48 views

How to discretize certain features with a feature set?

I am working with typing data with timing features(unit: ms) and some of the features are based on the keyboard keyCodes(positive integers, range:[8, 222]). Currently, I use ...
1
vote
1answer
54 views

Forecasting with a Machine Learning Algorithm

Im sorry if it is a too general question, but i am stuck somewhere between perfect and adequate in my model. So, i wanted to ask here. If it is not a suitable question, your negative feedbacks are all ...
3
votes
1answer
34 views

preprocessing time sequence

I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event. The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas: <...
1
vote
1answer
20 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
0
votes
1answer
11 views

Find differences of two protein sequence files

I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in ...
1
vote
1answer
56 views

How to control for Co-variate shift in test data set compared to train data for regression task?

I am working on a regression project. But I am facing the problem of covariate shift in features due to time delay.Test data was collected a year later due to which there has been some change in ...
2
votes
2answers
470 views

Pre-processing on MRI images

I have MRI images of brain tumors collected from a hospital (not a benchmark dataset). And I am planning to use them to predict/classify tumour types using a typical machine learning approach: texture ...
1
vote
0answers
17 views

What would be the best way to combine several denoised images and achieve superior denoising results? [closed]

I have generated denoised images using several models and would like to create an ensemble of these individual images to achieve superior denoising results. What would be the best way to combine (...
2
votes
1answer
142 views

Clustering time series based on monotonic similarity

Context I am involved in a task of clustering 1500 time series of 500 observations into a few number of clusters. The time series share all the same observed property at different spatial locations, ...
1
vote
1answer
6k views

“ValueError: Index contains duplicate entries, cannot reshape” error when I try to use pd.MultiIndex.arrays

I have data which includes id , gender , collected time test name and Test values , Units of measurement Test Names will include all tests that a patient taken and Value col will have its ...
2
votes
1answer
31 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...

1
2 3 4 5
8