Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [normalization]

The tag has no usage guidance.

0
votes
0answers
17 views

Should I concatenate one-hot vectors and real vectors as input feature?

I have a set of input features consisting of the following for each row of data: real vectors (1x128 dimensions, between [1,1000000000] ) one-hot vectors ( 1x168 dimensions, i.e. 7 days 24 hours ) ...
0
votes
2answers
15 views

Keras y-labels range between 0 and 1 instead of binary?

I have X values and corresponding y-labels, until now I used to round my labels <0.5 to 0 and >0.5 to 1. Is it possible to use values between 0 and 1 for "y train"? Using Keras and Tensorflow. ...
0
votes
1answer
40 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
0
votes
0answers
15 views

consine similarity results are high

I have vectors of length 160, and I'm trying to measure the cosine similarity between they. Each value in the vector represents a feature (frequency of words in lexicons), so most (not all; to give ...
1
vote
1answer
68 views

SVM model classifying into one class only, after standardization

I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are ...
1
vote
0answers
8 views

Word frequencies in unbalanced case-control dataset

I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, ...
2
votes
2answers
51 views

How can I augment my image data?

What are the correct and common ways to normalize image for CNN? I used to work with text and it was pretty straightforward. Removing stop words, clean text from noise, tokenization, stemming etc. ...
0
votes
0answers
15 views

Normalize between -1 and 1 or 0 and 1 (for LSTM)

Looking at various examples on the Internet I see some people normalize between -1 and 1, and others between 0 and 1. Is there any reason people choose one over the other? Assuming I'm using the ...
1
vote
1answer
19 views

normalization of probabilities in predicting a poly-neuron output in neural nets

When predicting a poly-neuron output in neural nets, say, predicting multiple handwritten digits and giving an output neuron vector (0.1,...,0.9,0.1,...), many use sth like softmax (or sth like the ...
0
votes
0answers
23 views

Bimodal distribution after log-transformation

I'm trying to predict house prices from a given dataset. Since the distribution of the target variable was skewed: I've done log-transformation: And got a somewhat bimodal distribution. What should ...
0
votes
1answer
25 views

PANDAS Within Category Normalization

I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this: ...
1
vote
1answer
24 views

Is it better to use a MinMax or a Log Return normalization to predict stock price movements?

I am trying to use a LSTM model to predict d+2 and d+3 closing prices. I am not sure whether I should normalize the data with a MixMax scaler (-1,+1) using the log return (P(n)-P(0))/P(0) for each ...
0
votes
1answer
29 views

How to normalize data of a different nature?

I am working a price prediction LTSM model for the stock market. I am using multiple features: Open, Close, High and I would like to add the Volume. The 3 first features are of the same nature but ...
1
vote
2answers
56 views

Why normalize when all features are on the same scale?

So I'm doing the tensorflow tutorial found here: https://www.tensorflow.org/tutorials/keras/basic_classification Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector. ...
0
votes
0answers
8 views

How to denormalize data used to train a RNN?

I am running the code defined in the "usage" section of this repo. I get the same output : Before training and testing, the data is normalized with the following code : ...
0
votes
0answers
18 views

normalize output of a feed-forward ANN

My feed-forward neural-network is modeling (regression) a multi-channel loss function. The output of the network is a vector y (size 10) that describes the loss ratio of the input signal x for each ...
2
votes
2answers
88 views

normalizing data and avoiding dividing by zero

I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly ...
0
votes
1answer
27 views

Normalizing / standardizing training and validation data

Say I split my data to 80% training and 20% test/validation and I want to standardize it, I think I'm right in saying I shouldn't standardize across 100% of the data, and then do the split, because ...
1
vote
1answer
18 views

Normalization set data

There are several methods to normalize data, among them are: min-max, z-score and scale decimal. Can I use any one or with what criteria should I choose one of them?
0
votes
1answer
89 views

MinMaxScaler returned values greater than one

Basically I was looking for a normalization function part of sklearn, which is useful later for logistic regression. Since I have negative values, I chose MinMaxScaler with like so: ...
7
votes
1answer
181 views

What are some situations when normalizing input data to zero mean, unit variance is not appropriate or not beneficial?

I have seen normalization of input data to zero mean, unit variance many times in machine learning. Is this a good practice to be done all the time or are there times when it is not appropriate or not ...
4
votes
3answers
99 views

Understanding data normalisation

So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should ...
0
votes
0answers
20 views

Adding and Normalizing extra features to Word2Vec representation

My problem is kind of similar to this question I am currently using a word2vec 100 features representation of my words. However, I want to add more features to have more similarity between synonyms ...
2
votes
1answer
297 views

Batch Normalization and Input Normalization in CNN

I build my CNN on Keras, normally in the ImageDataGenerator I saw the rescale = 1. / 255 used to normalize input data (pixel ...
0
votes
1answer
170 views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
1
vote
2answers
27 views

How to standarize feature vector with data in different scales?

Let's suppose I have a dataset with numerical attributes of different types. Let's suppose I want to employ a Neural Network for supervised classification with that dataset. For that, I need to ...
1
vote
1answer
37 views

It is helpful to normalize target variables for a regression neural network?

It is customary to normalize feature variables and this normally does increase the performance of a neural network in particular a CNN. I was wondering if normalizing the target could also help ...
1
vote
0answers
37 views

Regression Decision Tree - Normalize or Split into Ranges a continuos feature

I have in my dataset a feature named distances which ranges goes from 200 to 12000 (more or less). Since the other features have got values under 50 I need to do ...
0
votes
1answer
18 views

How to provide classified feature to a neural network

Let say I have a feature that may have one of 4 values, 1,2,3,4. I want to provide it as a NN input, what is proper way to do that? I can map it like 1 -> -1.0 | 2 -> -0.3 | 3 -> 0.3 | 4 -> 1.0 , or ...
3
votes
1answer
88 views

Normalizing test data

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the ...
1
vote
5answers
160 views

When should I normalize data?

I often see that numeric values in machine learning is scaled to 0-1 range. Why is it better? I have some temperature values in my training set. What if I will have some values to predict that will ...
0
votes
1answer
212 views

Subtracting grand mean from train and test images

I am building an image classifier based off the VGG_face keras implementation. It is easiest for me to extract a csv file full of the representations and then try classifiers on those representations. ...
2
votes
1answer
318 views

Data scaling before PCA: how to deal with categorical values?

I have to apply PCA on a dataset, which contains both numerical and categorical values. In the preprocessing phase, I converted all the categorical values in numerical, so that the software can deal ...
2
votes
3answers
60 views

How to properly mean-center my data

My task is to estimate a person's age based on a rgb image of the face of that person. I'm using ResNet-50 to that end. At first stage I trained my net on a dataset which is called WIKI-IMDB (after ...
0
votes
1answer
23 views

Should I normalize data to be similar to Normal distribution?

As I understood, when training a neural network, it is preferable to have data with expectation of 0 and std. of 1. Now if I have a feature with a Ratio distribution i.e., where median and ...
1
vote
0answers
24 views

How to preserve levels with Z-score?

I normalize time series data using a sliding window to be fed into a Neural Net. I normalize each window independently using Z-score. (Because on the entire dataset proven to be useless. Because on ...
1
vote
0answers
16 views

How to pre-process frequency of a series of signals?

I use a neural net to generate predictions based on a time series of signals. I use a sliding window to feed the data to an LSTM model. The input signals have a random frequency that - I believe - is ...
2
votes
1answer
234 views

How to normalize a boolean feature for neural nets?

I have a feature that is boolean and I would like to feed it to a neural net as one of the inputs. I think in theory the best is to encode as false->0 and true->1 because 0 as an input will deactivate ...
3
votes
2answers
38 views

Normalising data with multiple methods

When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score ...
9
votes
3answers
2k views

Zero Mean and Unit Variance

I'm studying Data Scaling, and in particular the Standardization method. I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit ...
2
votes
1answer
239 views

Using z-score for neural network normalization

I've read many people use z-score to normalize their data for presenting to a neural net, and that all data should lie in a range (usually -1 to 1), but z-score can return results beyond those bounds. ...
4
votes
3answers
807 views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
1
vote
1answer
224 views

How to Normalize & Scale a Single Data Point

I do understand the concept of normalizing & scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms....
2
votes
2answers
285 views

Why not use Scaler.fit_transform on total dataframe?

In sklearn I'm normalizing the data with MinMaxScaler. The example I'm following uses ...
2
votes
0answers
25 views

Normalising time(minutes) related data with n other input variables(also dependant on time)

I am working on the GPS dataset of the football players with the energy expenditure being my output variable and other 15 variables being input/independent variables. My objective is to forecast the ...
0
votes
1answer
63 views

De-normalization in Linear Regression

I have implemented a Linear regression model on a dataset of 7 independent variable and 1 target with the below 2 approaches 1) Without normalization of the data, resulted in a Mean squared error of ...
1
vote
1answer
126 views

What is difference between detrend and normalization? [closed]

matlab function detrend subtracts the mean from data. If data contains several data columns, detrend treats each data column separately. One of the normalization technique is subtracting the mean ...
2
votes
3answers
2k views

How do we standardize arrays with NaN?

I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there ...
1
vote
1answer
65 views

Standardization and Normalization

Which and all Machine Learning algorithms needs the data to be standardised/normalised before feeding into the model. How do we determine whether the particular model/data needs to be standardised/...
0
votes
0answers
77 views

The correct way for pre-processing of Time series rnn

I have a question regarding normalization for time series data before passing RNN network. Is this will improve the accuracy of the model? I tried stock price with and without normalization, and it ...