3
$\begingroup$

I had a simple neural network that was outputting the same value regardless of the input. During training, it was behaving normally, with training and validation loss diminishing to a floor value.

The data range was in [-1000, +1000]. The model is a 1D convolutional network. If it is useful :

    train_loss = lcategorical_crossentropy(...).mean()

    net = InputLayer((None, net_z, net_x), input_var=input_var)

    net = ConvLayer(net,  16, 9, pad='same' )#, flip_filters=False)
    net = ConvLayer(net,  16, 9, pad='same')#, flip_filters=False)

    net = DenseLayer(net,   num_units=32)
    net = DropoutLayer(net, p=0.5)

    net = DenseLayer(net, num_units=2, nonlinearity=None)
    net = NonlinearityLayer(net, softmax)

This was corrected by normalizing the data in [-1, 1]

What is the reason behind that ? Backpropagation being stuck I guess, but where can I learn about that in depth ?

$\endgroup$
1
$\begingroup$

If it works for your data, I would suggest normalization by: input = (input-median(input)) / std(input)

Also try to use a smaller batch size for initial training. And triple check how your data actually looks like. The same goes for the labels.

Depending on the framework, you should also check the data type of the labels. Some unfortunate computation might cast your float probabilities to int, thereby killing any meaningful output.

I can recommend Andrew Ng's famous Coursera lectures. They give good insight into what happens within the neurons.

$\endgroup$
  • $\begingroup$ The network is working fine after normalization (no need for data-labels validation), was behaving correctly during training (gradient not stuck), and single output was present regardless of the size of the batch. What I want to understand is why it did not perform well before normalization. $\endgroup$ – bold Nov 17 '16 at 10:03
  • $\begingroup$ extreme inputs cause trouble with $\endgroup$ – michaelosthege Nov 18 '16 at 12:37
  • $\begingroup$ (dammit) extreme inputs cause trouble with (assuming tanh actiation) * gradients (because a sigmoid activation is essentially flat at large z) * with regularization, parameters will still have a gradient pushing them towards 0 which counteracts the previous problem but leads to the following: * floating point precision (really small parameter multiplied with large input in order to put it in a reasonable range for preactivation) The network might still use the biases to adjust the prediction towards the mean label (decreasing the loss) but the other weights get stuck $\endgroup$ – michaelosthege Nov 18 '16 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.