To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for training and got excellent results. So I wonder if CNNs aren't necessary for this purpose if a double layer fully connect network can do the job well?! For example, I'd like to recognize digit '1'. I use only 4 or 5 images that resemble '1', and other 4 or 5 images that look like anything else. Every time I take~2000 pixels from each image. It turns out this code with this tiny amount of training data can recognize correct and incorrect digits really well.
You might be able to get pretty good results on a simple task, but the fact of the matter is that taking random pixels (or indeed just flattening out ALL pixels) essentially destroys any structural information that was contained within the original image.
This was the insight behind convolution networks (from original author Yann LeCun), as they really find areas of correlation in the position/structure of the image input across the dataset. So they understand, for example, the a "1" offers high correlations of pixels in a vertical line, normally in the center of the image input.
This information is no longer included with randomly selected pixels and has been almost destroyed by flattening all pixels into a single vector.
If your use-case requires a certain accuracy and you are reaching that with your simple neural network (or otherwise), then of course it is perfectly valid and you can be happy :-)