# Convnet training error does not decrease

I'm training a convoluted neural net to drive a toy car, and no matter what I do the training accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?

Notes:

• Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records.
• Three equally proportioned classes: turn left, go straight, turn right
• Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6
• Simple NN uses the sigmoid activation function, the convnet is all relu activations
• Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs
• My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion
• Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%
• I'm using initial = tf.truncated_normal(shape, stddev=0.1) to initialize all weights
• I'm using initial = tf.constant(0.1, shape=shape) to initialize all biases
• It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy

Convnet Notes (all yield same poor results):

• Batch normalization
• Max pooling
• 50% drop-out probability
• Various stride sizes
• Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons.
• 3D convolution (took too much memory and caused my GPU to crash)
• 1x1 convolutions
• The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine

Code:

• What happens if you keep training on one instance, will it overfit on this sample? Jun 12, 2017 at 8:58
• Don't compute cross entropy that way: github.com/tensorflow/tensorflow/issues/2462 Jun 14, 2017 at 13:36
• "DO NOT use this code for training. Use tf.nn.softmax_cross_entropy_with_logits() instead." Jun 14, 2017 at 13:37
• @JanvanderVegt The training accuracy doesn't improve on the same batch of 50 images, so something is clearly wrong. The single layer neural network is able to overfit when fed on the same single batch data though. Jun 17, 2017 at 4:31
• @kbrose Your suggestion did the trick! I tried it out and my convnet is now comparable to my fully-connected net. If you write up your answer I'll accept it. Jun 17, 2017 at 23:41

In your convnet code, you compute the cross entropy manually:

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))


If you do that, you may run into numerical stability issues. Instead, you should use

tf.nn.softmax_cross_entropy_with_logits()


If using tensorflow version 1.8 or later, use

tf.nn.softmax_cross_entropy_with_logits_v2()


and weep at the state of tensorflow's API.