I try to train convolutional neural net for a classification problem. However, my neural net is not learning anything. It guesses only two labels and ignores the rest. Even though I try to train to overfit my neural net, the loss function is not decreasing at all. My I try to make my network go as deep as 12 layers of the convolutional neural net in order to overfit the subsampling data. However, it is not working. Do you have any suggestion what to look?
Make sure that while training, you are passing the data in a random order. It looks like you are passing all images of one class at a time then the next class etc. Shuffle the images and pass it in a random manner.
There are a bazillion things that can go wrong when training a DNN, and it's often helpful to heavily simplify your starter net/data before you move onto your actual problem.
Without any more information on the specifics of your setup, etc, the first thing I would recommend for you to try is to overfit your net to random noise. Don't make it deep just yet! You want to make sure that everything else if your setup is kosher first.
So: Create a random-noise data set with labels, OR extract a small (< 50) or so images for you to use as your dev set, and try to overfit on those in the very beginning, with a vastly simplified net. This should always be your first step.