So I was training a fairly shallow convnet, because my deepnet based on vgg19 wasn't working. 2 conv layers and 2 flat layers, the second flat layer was the output.

It converged quickly to all zeros in the second conv layer. First conv layer wasn't all zeros. First flat layer appears to have learned the ditribution of classes.

So it appears that my network's strategy os to ignore the inputs and just predict the class distribution. I tried class weights, same result just a more uniform distribution gets predicted.

I changed the learning rate, optimizer, and even tried gradient clipping. I augmented the data, and introduced regularization on the data, and network via dropout layers. No luck. Same result.

Why would my network exhibit this behavior and what can be done about it?


Here is a code snippet that defines the network that I am using:

def discriminator_model():

model = Sequential()
                 (10, 10),
                 input_shape=( 256, 256, 3),
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dense(228, activation='sigmoid'))
return model

After training I was trying to see what the layers had actually learned with the following code:

from keras import backend as K

inp = d.input                                           # input placeholder
outputs = d.layers[3].output
functors = K.function([inp], [outputs])
arr = functors([x])[0].reshape(64,29,29,1)
for i in range(32):
     #p = (255./(np.max(arr[i])-np.min(arr[i])))*(arr[i]-np.min(arr[i]))

Here is an example of what I am getting as output from the output from 2nd conv layer

And here is example output from the first layer for comparison, which I generated with similar code: enter image description here

Also tried changing loss functions, removing pooling layers, etc. I know it is weird. I've never come across anything quite like this before.

  • $\begingroup$ Is your data normalised? i.e. each feature has zero mean, and unit variance? $\endgroup$ – timleathart May 3 '18 at 6:29
  • $\begingroup$ can you share the code snippet here? $\endgroup$ – Jason Ray May 3 '18 at 10:59
  • $\begingroup$ What data domain are we talking about? VGG and conv nets in general are geared towards images, but I find the scenario hard to believe if you are indeed trying to classify images. As @timleathart said, normalising input might be helpful. You could try using batch normalisation. $\endgroup$ – n1k31t4 May 3 '18 at 11:08
  • $\begingroup$ I am classifying images into 228 categories, one image can fall into multiple categories. The data wasn't normalized to mean zero and unit variance but it was scaled to be between zero and one. I'm adding some code snippets to the question. $\endgroup$ – Ryan May 3 '18 at 14:54
  • $\begingroup$ Re scale back your images to coloured regions.. Have a look at this kernel of mine which demonstrates what you are looking for....(with few modifications for your problem) kaggle.com/adityaecdrid/mnist-with-keras-for-beginners-99457 $\endgroup$ – Aditya May 4 '18 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.