0
$\begingroup$

I'm building a convoluted neural network to teach a toy car, powered by a Raspberry Pi, how to drive based on incoming streams of frames from a webcam mounted on top of the car. The top half of each image is irrelevant. What matters is the curvature of the road, and this is in the bottom half.

I've generated a substantial amount of data (about 40k records) by driving the car around myself and recording what I do (commands are left, right, and straight) and what the frames are. However my trained ConvNets aren't giving me the performance I'd hoped for. My experimentation with one of the deployed models confirms that it is indeed tricked by changes in the top half of the streaming images.

A simple solution is to programmatically cut the frames in half so that the neural net only receives relevant portions. However, deep neural nets are praised for their ability to learn features with (ideally) zero human future transformations, so I want to avoid this approach. This project is meant to be a learning experience for me so I want learn how to architect the ConvNet more effectively. Each training session takes several hours to run, so rather than try everything under the sun, I'd figure I'd reach out to the community here to narrow my focus of exploration.

One thought I have is to put a fully connected (FC) layer in front so that subsequent layers effectively only convolve over the relevant portion of the image. I think this could potentially work if this up front FC layer learned to assign very small weights to pixels in the top half of the image. Could this work? Are there better architectures?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You could be right that ignoring top part of image would benefit the CNN. However, there is very little point in trying to architect this - if your premise that the CNN will ignore irrelevant details in the top half is correct, then that will occur anyway and there is no standard NN architecture that will help that other than disconnecting the top half of network, which is going to be logically exactly the same as programmatically slicing the image, with the disadvantage of storing and calculating with twice as many parameters.

You should either programatically cut the image in half or do nothing to the image and rely on the CNN's inherent ability to give low weights to irrelevant details. If you do the latter, you may be able to get around the learning of incorrect details in the top half by augmenting your data - e.g. add some noise to images*, especially in the irrelevant top half. Perhaps horizontally flipping a few images (and reverse relevant targets for the control class) might be another useful augmentation. Some augmentations could also be useful if you just take the lower half of the image.

* Noise should be something close to variations that could be seen when in use. E.g. slurring pixels left or right might be reasonable. Inserting "static" probably is not.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So is it safe to say that a convoluted layer won't output activations in a way that distorts the orientation of the image? And that a fully connected layer at the end should be capable of differentiating top-half convolutional activations from bottom half convolutional activations? If the orientation doesn't get lost, I could see how the network should be able to learn from unaltered images $\endgroup$
    – Ryan Zotti
    Jul 10, 2016 at 20:26
  • $\begingroup$ Multiple convolutional layers with max pooling or strided convolution tend to lose resolution - of where detected features are exactly - as they get deeper. But they don't otherwise distort things. So the feature maps from the last convolutional layer should still effectively separate features detected in top of image from same features detected in the bottom. $\endgroup$
    – Neil Slater
    Jul 11, 2016 at 5:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.