I'm training a convoluted neural net to drive a toy car, and no matter what I do the training accuracy does not increase beyond 30-35%, which is where it starts when the convnet is randomly initialized. What's strange is that a much simpler model, a neural network with a single hidden layer and no convolution, does substantially better, consistently getting accuracy of 65-75%. I've been working on this project for over a year and feel like I've tried everything to make the convnet better. What am I doing wrong?
- Dataset contains 150,000 records, but since my video feed generates 20 frames per second and since frames don't change much from second to second, it's more like 150,000 / 20 = 7,500 unique records.
- Three equally proportioned classes: turn left, go straight, turn right
- Both the convnet and simple NN use Tensorflow's AdamOptimizer. The simple net does well with 1e-5 and 1e-4, but the convnet doesn't do well with any value, I've tried 1e-2 all the way through 1e-6
- Simple NN uses the sigmoid activation function, the convnet is all relu activations
- Both models read the same data. I have a single sampling and data augmentation class that's shared among all my models, so my data isn't bad because the simple net works fine on the same inputs
- My convnet can't even overfit on the training data, so it's not data size that's the issue, in my opinion
- Overfitting doesn't seem to be a problem with either model: training and validation sets tend to have similar performance, give or take 5%
- I'm using
initial = tf.truncated_normal(shape, stddev=0.1)to initialize all weights
- I'm using
initial = tf.constant(0.1, shape=shape)to initialize all biases
- It could be specifically the convolution that's the problem, since a shallow single-hidden-layer convnet did poorly but a two-hidden-layer fully-connected neural net with no convolution gets around 65% accuracy
Convnet Notes (all yield same poor results):
- Batch normalization
- Max pooling
- 50% drop-out probability
- Various stride sizes
- Various depths of layers: I've tried 2-5 convent layers, multiple 2-4 fully connected layers. I've had as few as 3 layers and as many as 9. Convnet layers have between 32 and 64 neurons, fully connected layers have between 32 and 512 neurons.
- 3D convolution (took too much memory and caused my GPU to crash)
- 1x1 convolutions
- The one thing I haven't tried is transfer learning, and I hope to only use that as a last resort since the simple net works fine