5
$\begingroup$

All the techniques/models that I have learnt so far for deep learning start with some sort of normalization to the features, for example gaussian method, minmax scaling, robust scaling, batch normalization, instance normalization.

Are there any techniques to run neural networks without normalization so that the network can see (in absolute values) the magnitude of the value and respond according to that instead of normalized values? Will there be exploding/vanishing gradient issues if I don't normalize my data?

For example, if I am training a custom LSTM network for multivariate time series data, the input dimension for a feature vector $x$ is all the values from $t-n$ to $t$, where $n$ is the number of time steps and the output vector is the value at $t+1$. Is there any need for normalization in this case?

| improve this question | |
$\endgroup$
  • $\begingroup$ normalizing data is not necessary, it just accelerates the speed of learning process. So you can use neural nets without normalizing. Also there are some struggles which show that batch normalization for example, is not suitable for all learning cases. For probable answers, would you please edit the second paragraph. It is a bit hard to be interpreted. $\endgroup$ – Media Dec 7 '17 at 21:39
3
$\begingroup$

Normalization helps to eliminate scale factors that might exist between variables in your data. Take, for example, the classic problem of predicting home prices. If you represent the square footage of your home in square millimeters, a large change in this value will have a relatively small effect on home price, implying a small gradient on this variable. If you represent that value in square kilometers, a small numerical change will have a large impact on price, implying a large gradient. Normalization isn't necessarily required, but can help to balance the problem by making all variables have "equal weight" in your model. If you were to include both the square millimeter and square kilometer variables in your training data, the neural network would likely spend a lot of effort optimizing on the square kilometer variable, since it is numerically more important. You can still do training with un-normalized data, but it will likely take longer, and possibly have worse output if your important variables are numerically smallest.

| improve this answer | |
$\endgroup$
  • $\begingroup$ What if the output feature vector is same as the input feature vector. So if I am training a custom LSTM network for time series data, the input dimension for a feature vector 'x' is all the values from 't-n' to 't' where 'n' is the number of time steps and the output vector is the value at 't+1'. Is there any need for normalization in this case ? I have also added this example to my question. $\endgroup$ – Ravi M Dec 7 '17 at 22:09
  • $\begingroup$ Can't comment so I pasted an answer. Please don't down-vote. I can confirm Nuclear Wang's answer that the results could be worse. But I would say, it's quite possible that you'll never get any good results when you have both big scale data with small scale data and both of them are of physical meanings. In my case, I actually have two features, one range from 0 to 2 and another range from -1 to 2e-5. I was thinking the value is actually not so different but it turns out that traditional loss functions are trying to optimize the 0-2 feature most of times and the 2e-5 parts are ignored for L1LOS $\endgroup$ – user2189731 May 7 '18 at 6:48
  • $\begingroup$ In case of real-valued inputs, you want to predict t+1 sample given previous t samples, use relative value instead of absolute ones, also normalize the differences to have zero mean unit Gaussian, because if you don't do so, the network may overfit. see arxiv.org/pdf/1308.0850.pdf for more intuition $\endgroup$ – Fadi Bakoura May 7 '18 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.