I had a simple neural network that was outputting the same value regardless of the input. During training, it was behaving normally, with training and validation loss diminishing to a floor value.
The data range was in [-1000, +1000]. The model is a 1D convolutional network. If it is useful :
train_loss = lcategorical_crossentropy(...).mean() net = InputLayer((None, net_z, net_x), input_var=input_var) net = ConvLayer(net, 16, 9, pad='same' )#, flip_filters=False) net = ConvLayer(net, 16, 9, pad='same')#, flip_filters=False) net = DenseLayer(net, num_units=32) net = DropoutLayer(net, p=0.5) net = DenseLayer(net, num_units=2, nonlinearity=None) net = NonlinearityLayer(net, softmax)
This was corrected by normalizing the data in [-1, 1]
What is the reason behind that ? Backpropagation being stuck I guess, but where can I learn about that in depth ?