# Several fundamental questions about CNN

I am trying to make a CNN for 3D image recognition but everything is predicted to only one class out of three. And the prediction even quickly converges during the first epoch. I have been working on this for an week and totally lost.
I have my own several guess why it always converges to one class.

1. My 3D tensor is huge as 40x35x30 and there are a lot of white spaces because I had to put my objects in cubic box. Would this be problematic?
2. I have only ~5000 samples for training and ~500 for test sets. Do I have too little samples?
3. My labels have 3 classes and ~75% of the whole data belong to the class number 1. During the first epoch, my model quickly converges to predict 99% of data as this class. Would this unbalanced data cause the problem?
4. My model is designed as
conv1(kernel_size=10,stride=1,filter=32)-batch_norm1-maxpool1-conv2(8,1,64)-batch_norm2-maxpool2-conv3(8,1,64)-maxpool3-fc4(1024)-batchnorm4-dropout4(20%)-fc5(384)-batchnorm5-dropout5(20%)-fc6(3)
I standardized the input between -1 and 1 I use leaky-relu activation for conv layers. I use Adam optimizer with decay rate of 0.99.
First, I am not sure if it is okay to perform batch normalization at every layer. Do I miss any important concept for designing CNN model here?
5. or maybe my data is just bad.. I kind of suspect that my data are not significantly different from each other and there is no pattern at all. In this case, is there any statistical method/model to check if my data have meaningful differences? (The 3D images are some chemical/physical data in 3D space that I converted as numpy matrix)

But I think it is more like vanishing problem because when I initialize the variable with Xavier's way, the convergence to the class #1 is slower. Please someone help me :(

## 2 Answers

1. You more than likely do not have enough training data for a neural network.
2. Your class imbalance problem is probably an issue. Instead of using accuracy as a measurement trying some type of F-score.
3. Batch normalization should be applied between the convolution layer and the activation function.
4. If you think you have a vanishing or dying activation problem, plot the gradients or the sum of gradients. It'll give you an idea if you're right or not.

Here's my take on each of your points.

1. You have very sparse data. Are you storing these data optimally in a sparse object, for example a csr sparse matrix if you're using Keras? This is more likely to affect training time than accuracy, but is something to think about.
2. 5000 training examples and 500 testing examples sounds alright to me, except you may have too few data to fit the type of model you have (you may be severely underfitting). Try a simpler model and see if you can improve results with that (try something stupidly simple like an MLP with one layer and see what happens).
3. This seems the most likely problem. My first guess is this has to do with the unbalanced classes. If you're using a metric like accuracy to evaluate training at each epoch, I'd recommend instead using something like average_precision or roc_auc_score or f1, all available through scikit-learn. If you're using Keras, try using class_weights as well, which will weight underrepresented classes higher in the loss function, essentially biasing the algorithm to consider underrepresented classes on an equal playing field. If you're not using Keras, try implementing some similar class weighting scheme.
4. Batch normalization is less popular now than it used to be. It's worth trying the architecture with and without batch normalization to see if it provides a clear benefit.
5. Yes, if your data are purely random then you will not detect a signal at all. This goes back to point number 2 (try a simpler model and see if you can detect a pattern). You can also try visualizing some of your samples to see if you can visually see a difference between the classes.

With regard to Xavier initialization, I don't think that's the likely cause of the effect you're seeing. The type of initialization can affect results, of course, but from what you're describing, I strongly suspect this is due to too little signal from the data or the unbalanced classes problem.

Hope that helps!

• Thanks! I am using TensorFlow and after using tf.nn.weighted_cross_entropy_with_logits instead of tf.nn.softmax_cross_entropy_with_logits it does not predict everything as a specific class. But can roc_auc_score or f1 be used as loss function? Or did you just suggest those so I can judge the result better? – huddy Jan 30 '18 at 10:01
• Hi @huddy, no, roc_auc_score and f1 are not loss functions - they're metrics. A loss function is something like cross-entropy (for classification) or mse (for regression). A metric is a separate way to evaluate your model. For example, your cross-entropy loss may be very low if in your training data you have many examples of class 1 and few examples of class 2, and your model assigns all examples to class 1; however, your roc_auc_score may be low in that case. So yes, I suggested those to enable you to better evaluate your model's results. – StatsSorceress Jan 30 '18 at 16:38
• again, thanks so much! I can finally proceed my project with your help. I have two more questions. Is there any tip to determine the weights for each class? The result is quite sensitive to how I set an weight for each class. Do I just keep trying different parameters until I am satisfied? And would you give any comment on the softmax cross entropy loss function that I am using with regard to the class imbalance? – huddy Jan 30 '18 at 19:42
• Hi, yes, you should look at the ratio of the classes in your training set. For example, if there are 100 examples of class 1 for every 1 example of class 2, then you should set the weights with that ratio. Something like this: class_weights = {0.:1., 1.:(class_zeros_train/class_ones_train)}. Also, yes, I'm using weighted_cross_entropy_with_logits for a similar problem. If my answer is helpful to you, please consider accepting it? – StatsSorceress Jan 30 '18 at 22:22