Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [normalization]

The tag has no usage guidance.

2
votes
0answers
9 views

Should one normalize the frequency values when feeding it as an input to machine learning model?

Consider an unsupervised data. The data is in the form of a csv file( I am using pandas dataframe for this). Its a web page data at different time steps and the way I am converting data to be fed to ...
2
votes
1answer
20 views

Why does test data need to be normalized on train data mean and std?

I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data. Why ...
0
votes
0answers
17 views

Normalizing Jaccard similarity scores in relation to differences in document length

The Jaccard similarity of two documents A and B can be defined as the size of their intersection (how many tokens are in both docs) divided by the size of their union (total number of tokens found in ...
0
votes
0answers
71 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
13 views

Image normalisation methods

I have found some research papers specifying explicitly the normalisation technique they used to get the results. What difference do IMG /255.0 And ...
0
votes
2answers
40 views

Why normalization kills my accuracy

I have a binary car sound classifier. I have a feature set that is extracted from audio with size of 48. I have a model(multi layer neural network) that has around %90 accuracy on test and validation ...
0
votes
2answers
29 views

How to normalize just one feature by scikit-learn?

Wanna apply a specific scaler, say StandardScaler, on a specific feature, keeping other features intact. the dataset format is something like: [ [1, 0.2, 1000], [2, 0.1, 2400], [3, 0.9, 7620] ] I ...
2
votes
1answer
50 views

Financial Time Series data normalization

I'm using Keras in R to predict financial time series. It's easy to normalize price, simply compute returns or log returns, usually it's enough. I want to use Goldman Sachs Financial Conditions Index ...
0
votes
0answers
44 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
0answers
11 views

Will mean-value imputation have the same effect in these two cases before normalization?

Will mean-value imputation have the same effect, if performed before normalization, on the distribution of the normalized values for those values that were originally not missing, for min-max ...
0
votes
0answers
26 views

Should I concatenate one-hot vectors and real vectors as input feature?

I have a set of input features consisting of the following for each row of data: real vectors (1x128 dimensions, between [1,1000000000] ) one-hot vectors ( 1x168 dimensions, i.e. 7 days 24 hours ) ...
0
votes
2answers
56 views

Keras y-labels range between 0 and 1 instead of binary?

I have X values and corresponding y-labels, until now I used to round my labels <0.5 to 0 and >0.5 to 1. Is it possible to use values between 0 and 1 for "y train"? Using Keras and Tensorflow. ...
1
vote
1answer
49 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
0
votes
0answers
16 views

consine similarity results are high

I have vectors of length 160, and I'm trying to measure the cosine similarity between they. Each value in the vector represents a feature (frequency of words in lexicons), so most (not all; to give ...
1
vote
1answer
79 views

SVM model classifying into one class only, after standardization

I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are ...
1
vote
0answers
8 views

Word frequencies in unbalanced case-control dataset

I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, ...
2
votes
2answers
80 views

How can I augment my image data?

What are the correct and common ways to normalize image for CNN? I used to work with text and it was pretty straightforward. Removing stop words, clean text from noise, tokenization, stemming etc. ...
0
votes
0answers
18 views

Normalize between -1 and 1 or 0 and 1 (for LSTM)

Looking at various examples on the Internet I see some people normalize between -1 and 1, and others between 0 and 1. Is there any reason people choose one over the other? Assuming I'm using the ...
1
vote
1answer
32 views

normalization of probabilities in predicting a poly-neuron output in neural nets

When predicting a poly-neuron output in neural nets, say, predicting multiple handwritten digits and giving an output neuron vector (0.1,...,0.9,0.1,...), many use sth like softmax (or sth like the ...
0
votes
1answer
41 views

PANDAS Within Category Normalization

I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this: ...
1
vote
1answer
60 views

Is it better to use a MinMax or a Log Return normalization to predict stock price movements?

I am trying to use a LSTM model to predict d+2 and d+3 closing prices. I am not sure whether I should normalize the data with a MixMax scaler (-1,+1) using the log return (P(n)-P(0))/P(0) for each ...
0
votes
1answer
57 views

How to normalize data of a different nature?

I am working a price prediction LTSM model for the stock market. I am using multiple features: Open, Close, High and I would like to add the Volume. The 3 first features are of the same nature but ...
1
vote
2answers
65 views

Why normalize when all features are on the same scale?

So I'm doing the tensorflow tutorial found here: https://www.tensorflow.org/tutorials/keras/basic_classification Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector. ...
3
votes
3answers
238 views

normalizing data and avoiding dividing by zero

I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly ...
1
vote
1answer
36 views

Normalizing / standardizing training and validation data

Say I split my data to 80% training and 20% test/validation and I want to standardize it, I think I'm right in saying I shouldn't standardize across 100% of the data, and then do the split, because ...
1
vote
1answer
21 views

Normalization set data

There are several methods to normalize data, among them are: min-max, z-score and scale decimal. Can I use any one or with what criteria should I choose one of them?
0
votes
1answer
217 views

MinMaxScaler returned values greater than one

Basically I was looking for a normalization function part of sklearn, which is useful later for logistic regression. Since I have negative values, I chose MinMaxScaler with like so: ...
7
votes
1answer
195 views

What are some situations when normalizing input data to zero mean, unit variance is not appropriate or not beneficial?

I have seen normalization of input data to zero mean, unit variance many times in machine learning. Is this a good practice to be done all the time or are there times when it is not appropriate or not ...
4
votes
3answers
139 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
0
votes
0answers
46 views

Adding and Normalizing extra features to Word2Vec representation

My problem is kind of similar to this question I am currently using a word2vec 100 features representation of my words. However, I want to add more features to have more similarity between synonyms ...
2
votes
1answer
398 views

Batch Normalization and Input Normalization in CNN

I build my CNN on Keras, normally in the ImageDataGenerator I saw the rescale = 1. / 255 used to normalize input data (pixel ...
0
votes
1answer
444 views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
1
vote
2answers
28 views

How to standarize feature vector with data in different scales?

Let's suppose I have a dataset with numerical attributes of different types. Let's suppose I want to employ a Neural Network for supervised classification with that dataset. For that, I need to ...
1
vote
1answer
70 views

It is helpful to normalize target variables for a regression neural network?

It is customary to normalize feature variables and this normally does increase the performance of a neural network in particular a CNN. I was wondering if normalizing the target could also help ...
1
vote
0answers
41 views

Regression Decision Tree - Normalize or Split into Ranges a continuos feature

I have in my dataset a feature named distances which ranges goes from 200 to 12000 (more or less). Since the other features have got values under 50 I need to do ...
0
votes
1answer
19 views

How to provide classified feature to a neural network

Let say I have a feature that may have one of 4 values, 1,2,3,4. I want to provide it as a NN input, what is proper way to do that? I can map it like 1 -> -1.0 | 2 -> -0.3 | 3 -> 0.3 | 4 -> 1.0 , or ...
3
votes
1answer
103 views

Normalizing test data

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the ...
1
vote
5answers
311 views

When should I normalize data?

I often see that numeric values in machine learning is scaled to 0-1 range. Why is it better? I have some temperature values in my training set. What if I will have some values to predict that will ...
0
votes
1answer
375 views

Subtracting grand mean from train and test images

I am building an image classifier based off the VGG_face keras implementation. It is easiest for me to extract a csv file full of the representations and then try classifiers on those representations. ...
3
votes
1answer
437 views

Data scaling before PCA: how to deal with categorical values?

I have to apply PCA on a dataset, which contains both numerical and categorical values. In the preprocessing phase, I converted all the categorical values in numerical, so that the software can deal ...
2
votes
3answers
64 views

How to properly mean-center my data

My task is to estimate a person's age based on a rgb image of the face of that person. I'm using ResNet-50 to that end. At first stage I trained my net on a dataset which is called WIKI-IMDB (after ...
0
votes
1answer
23 views

Should I normalize data to be similar to Normal distribution?

As I understood, when training a neural network, it is preferable to have data with expectation of 0 and std. of 1. Now if I have a feature with a Ratio distribution i.e., where median and ...
1
vote
0answers
26 views

How to preserve levels with Z-score?

I normalize time series data using a sliding window to be fed into a Neural Net. I normalize each window independently using Z-score. (Because on the entire dataset proven to be useless. Because on ...
1
vote
0answers
17 views

How to pre-process frequency of a series of signals?

I use a neural net to generate predictions based on a time series of signals. I use a sliding window to feed the data to an LSTM model. The input signals have a random frequency that - I believe - is ...
2
votes
1answer
337 views

How to normalize a boolean feature for neural nets?

I have a feature that is boolean and I would like to feed it to a neural net as one of the inputs. I think in theory the best is to encode as false->0 and true->1 because 0 as an input will deactivate ...
3
votes
2answers
44 views

Normalising data with multiple methods

When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score ...
9
votes
3answers
3k views

Zero Mean and Unit Variance

I'm studying Data Scaling, and in particular the Standardization method. I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit ...
2
votes
1answer
362 views

Using z-score for neural network normalization

I've read many people use z-score to normalize their data for presenting to a neural net, and that all data should lie in a range (usually -1 to 1), but z-score can return results beyond those bounds. ...
5
votes
3answers
1k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
1
vote
1answer
341 views

How to Normalize & Scale a Single Data Point

I do understand the concept of normalizing & scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms....