To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training and got excellent results. So I wonder if CNNs aren't necessary for this purpose if a double layer fully connect network can do the job well?! For example, I'd like to recognize digit '1'. I use only 4 or 5 images that resemble '1', and other 4 or 5 images that look like anything else. Every time I take~2000 pixels from each image. It turns out this code with this tiny amount of training data can recognize correct and incorrect digits really well.

| improve this question | | | | |
  • $\begingroup$ is your output is one neuron with output 0/1 or 10 neurons that output 0/1 ? $\endgroup$ – Jérémy Blain Oct 31 '18 at 10:34
  • $\begingroup$ Say my input pic data are all of n by n pixels. Then my input layer has n^2+1 neurons (considering y=kx+b, k is of n by n dimension, and b is a scalar). My output layer has 2 neurons: 0 or 1, meaning incorrect or correct. $\endgroup$ – feynman Nov 3 '18 at 4:55
  • $\begingroup$ why 2 neurons ?? $\endgroup$ – Jérémy Blain Nov 3 '18 at 14:20
  • $\begingroup$ then how many should there be? $\endgroup$ – feynman Nov 5 '18 at 9:19
  • $\begingroup$ 10 ? because there are 10 digits ?? $\endgroup$ – Jérémy Blain Nov 5 '18 at 9:22

You might be able to get pretty good results on a simple task, but the fact of the matter is that taking random pixels (or indeed just flattening out ALL pixels) essentially destroys any structural information that was contained within the original image.

This was the insight behind convolution networks (from original author Yann LeCun), as they really find areas of correlation in the position/structure of the image input across the dataset. So they understand, for example, the a "1" offers high correlations of pixels in a vertical line, normally in the center of the image input.

This information is no longer included with randomly selected pixels and has been almost destroyed by flattening all pixels into a single vector.

If your use-case requires a certain accuracy and you are reaching that with your simple neural network (or otherwise), then of course it is perfectly valid and you can be happy :-)

| improve this answer | | | | |
  • $\begingroup$ Many thanks for sharing that seminal paper, and I'll have a read. $\endgroup$ – feynman Nov 3 '18 at 5:02
  • $\begingroup$ So far my code works fine, being able to recognize n numbers pretty well out of very few training data (less than 10 training data for n=2). The input layer contains all those pixels as neurons and the output layer has n neurons. Then I wonder under what circumstances will my code fail to work? $\endgroup$ – feynman Nov 3 '18 at 5:15
  • $\begingroup$ My simple linear regression takes into account no relative positional info across pixels. That's what you mean by 'taking random pixels (or indeed just flattening out ALL pixels)?' I turn all pics into binary (black and white) pics and their pixel matrices are all of 0 or 255 elements. One thing puzzling me is I'd better fix the background color of all pics to be either black or white, and hence the content (number) the other color. Later I relaxed that restriction, not worrying about it, but input more training pics of both sorts, then the code recognized both 'black' and 'white' numbers. $\endgroup$ – feynman Nov 3 '18 at 5:28
  • 1
    $\begingroup$ @feynman - Yes, I mean that inherent structure in the data is lost when you flatten the image into a single array. Regarding your 2nd point: it is possibly that your network learns to predict after being exposed to samples with both black and white pixels reversed. I would guess that a convolutional network would produce better results or reach the same results with less samples/training iterations. $\endgroup$ – n1k31t4 Nov 4 '18 at 0:13
  • 1
    $\begingroup$ @feynman - I would call it a fully connected network. The term Artificial Neural Network is a term that includes a wide range of networks; I suppose any network artificially modelling the network of neurons in the human brain. Furthermore, I wouldn't call yours a deep neural network, as that term is traditionally reserved for networks with 3 or more layers. $\endgroup$ – n1k31t4 Nov 7 '18 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.