Several fundamental questions about CNN

Question

I am trying to make a CNN for 3D image recognition but everything is predicted to only one class out of three. And the prediction even quickly converges during the first epoch. I have been working on this for an week and totally lost.
I have my own several guess why it always converges to one class.

1. My 3D tensor is huge as 40x35x30 and there are a lot of white spaces because I had to put my objects in cubic box. Would this be problematic?
2. I have only ~5000 samples for training and ~500 for test sets. Do I have too little samples?
3. My labels have 3 classes and ~75% of the whole data belong to the class number 1. During the first epoch, my model quickly converges to predict 99% of data as this class. Would this unbalanced data cause the problem?
4. My model is designed as
   conv1(kernel_size=10,stride=1,filter=32)-batch_norm1-maxpool1-conv2(8,1,64)-batch_norm2-maxpool2-conv3(8,1,64)-maxpool3-fc4(1024)-batchnorm4-dropout4(20%)-fc5(384)-batchnorm5-dropout5(20%)-fc6(3)
   I standardized the input between -1 and 1 I use leaky-relu activation for conv layers. I use Adam optimizer with decay rate of 0.99.
   First, I am not sure if it is okay to perform batch normalization at every layer. Do I miss any important concept for designing CNN model here?
5. or maybe my data is just bad.. I kind of suspect that my data are not significantly different from each other and there is no pattern at all. In this case, is there any statistical method/model to check if my data have meaningful differences? (The 3D images are some chemical/physical data in 3D space that I converted as numpy matrix)

But I think it is more like vanishing problem because when I initialize the variable with Xavier's way, the convergence to the class #1 is slower. Please someone help me :(

Answer 1

1. You more than likely do not have enough training data for a neural network.
2. Your class imbalance problem is probably an issue. Instead of using accuracy as a measurement trying some type of F-score.
3. Batch normalization should be applied between the convolution layer and the activation function.
4. If you think you have a vanishing or dying activation problem, plot the gradients or the sum of gradients. It'll give you an idea if you're right or not.

Answer 2

Here's my take on each of your points.

1. You have very sparse data. Are you storing these data optimally in a sparse object, for example a csr sparse matrix if you're using Keras? This is more likely to affect training time than accuracy, but is something to think about.
2. 5000 training examples and 500 testing examples sounds alright to me, except you may have too few data to fit the type of model you have (you may be severely underfitting). Try a simpler model and see if you can improve results with that (try something stupidly simple like an MLP with one layer and see what happens).
3. This seems the most likely problem. My first guess is this has to do with the unbalanced classes. If you're using a metric like accuracy to evaluate training at each epoch, I'd recommend instead using something like average_precision or roc_auc_score or f1, all available through scikit-learn. If you're using Keras, try using class_weights as well, which will weight underrepresented classes higher in the loss function, essentially biasing the algorithm to consider underrepresented classes on an equal playing field. If you're not using Keras, try implementing some similar class weighting scheme.
4. Batch normalization is less popular now than it used to be. It's worth trying the architecture with and without batch normalization to see if it provides a clear benefit.
5. Yes, if your data are purely random then you will not detect a signal at all. This goes back to point number 2 (try a simpler model and see if you can detect a pattern). You can also try visualizing some of your samples to see if you can visually see a difference between the classes.

With regard to Xavier initialization, I don't think that's the likely cause of the effect you're seeing. The type of initialization can affect results, of course, but from what you're describing, I strongly suspect this is due to too little signal from the data or the unbalanced classes problem.

Hope that helps!