6
$\begingroup$

I'm training a convoluted neural net to drive a toy car, and no matter what I do the training accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?

Notes:

  • Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records.
  • Three equally proportioned classes: turn left, go straight, turn right
  • Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6
  • Simple NN uses the sigmoid activation function, the convnet is all relu activations
  • Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs
  • My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion
  • Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%
  • I'm using initial = tf.truncated_normal(shape, stddev=0.1) to initialize all weights
  • I'm using initial = tf.constant(0.1, shape=shape) to initialize all biases
  • It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy

Convnet Notes (all yield same poor results):

  • Batch normalization
  • Max pooling
  • 50% drop-out probability
  • Various stride sizes
  • Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons.
  • 3D convolution (took too much memory and caused my GPU to crash)
  • 1x1 convolutions
  • The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine

Code:

Improve this question
$\endgroup$
6
  • $\begingroup$ What happens if you keep training on one instance, will it overfit on this sample? $\endgroup$
    – Jan van der Vegt
    Jun 12, 2017 at 8:58
  • $\begingroup$ Don't compute cross entropy that way: github.com/tensorflow/tensorflow/issues/2462 $\endgroup$
    – kbrose
    Jun 14, 2017 at 13:36
  • $\begingroup$ "DO NOT use this code for training. Use tf.nn.softmax_cross_entropy_with_logits() instead." $\endgroup$
    – kbrose
    Jun 14, 2017 at 13:37
  • $\begingroup$ @JanvanderVegt The training accuracy doesn't improve on the same batch of 50 images, so something is clearly wrong. The single layer neural network is able to overfit when fed on the same single batch data though. $\endgroup$
    – Ryan Zotti
    Jun 17, 2017 at 4:31
  • $\begingroup$ @kbrose Your suggestion did the trick! I tried it out and my convnet is now comparable to my fully-connected net. If you write up your answer I'll accept it. $\endgroup$
    – Ryan Zotti
    Jun 17, 2017 at 23:41

1 Answer 1

Reset to default
7
$\begingroup$

In your convnet code, you compute the cross entropy manually:

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-5).minimize(cross_entropy)

If you do that, you may run into numerical stability issues. Instead, you should use

tf.nn.softmax_cross_entropy_with_logits()

If using tensorflow version 1.8 or later, use

tf.nn.softmax_cross_entropy_with_logits_v2()

and weep at the state of tensorflow's API.

See this issue for more information: https://github.com/tensorflow/tensorflow/issues/2462

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ It made a huge difference. My deep convnet is now substantially better than single layer neural network. It's my best performing model by far. $\endgroup$
    – Ryan Zotti
    Jun 18, 2017 at 20:59
  • 1
    $\begingroup$ Very useful, but this function is deprecated now. Use insted tf.nn.softmax_cross_entropy_with_logits_v2 . $\endgroup$
    – Federico Caccia
    May 21, 2018 at 19:09
  • $\begingroup$ @Media I rolled back your edit. Just saying that it is "Issue 2462" is not meaningful because the reader has no context. Even if they were able to infer it was a reference to some GitHub issue, how would they know they were being pointed to an issue on the tensorflow repo? If you really want to shorten the link, please find a clearer way to do so. But I'm not sure why it's necessary. $\endgroup$
    – kbrose
    Jul 30, 2018 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.