1
$\begingroup$

Right now I'm training a deep neural network for a binary classification problem, with a feature set of winrates. As such, each winrate is bigger or equal to 0 but smaller than 100.

I've been getting promising results without normalizing the input data, until I normalized it and got staggeringly worse accuracy.

The input feature is a 2d matrix of size 20, and the network has four layers with differing numbers of nodes in each layer. I'm using sgd optimizer and ReLU activation for the hidden layers, and the softmax activation function for the output layer.

The thing I'm wondering is why I'm getting better results with the neural network without the normalization? Is it because the optimal hyperparameters required for the network with the normalized input are different from when it is not normalized?

$\endgroup$
1
$\begingroup$

The thing I'm wondering is why I'm getting better results with the neural network without the normalization?

From a theoretical standpoint, normalizing your input shouldn't affect accuracy, in a sense that a NN can converge to any range of input values.

The reason why we normalize our data when using NNs is that it offers some useful properties, mainly regarding convergence and convergence speed. You can read more details here.

Is it because the optimal hyperparameters required for the network with the normalized input are different from when it is not normalized?

Yes, I think that is your problem. Some hyperparameters might be optimized for the regular input and might not work well for the normalized one. An example of this could be a higher learning rate, which might work for a higher range of input values but not for a normalized one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.