5
$\begingroup$

An 1024*1024 pixel image has around one million pixels. If I would like to connect each pixel to an R,G,B input neuron, then more than 3 million neurons are needed.

It would be really hard, to train a neural network, which has millions of inputs. How is it possible, to reduce the number of neurons?

$\endgroup$
  • $\begingroup$ A million inputs is not considered that big anymore, but for argument's sake, you could use an alternative basis such as wavelets, DCT, SVD, NMF, etc. Alternatively you could downsample; does the learning task really require full resolution? $\endgroup$ – Emre Dec 27 '15 at 17:52
  • 1
    $\begingroup$ @Emre Really? Please give me reference to 3 papers which make use of input vectors of a dimension of at least 5 million to substantiate your claim. $\endgroup$ – Martin Thoma Dec 28 '15 at 12:02
  • $\begingroup$ Pick any paper with categorical variables. NLP is a good place to look. If you want to represent every word in the language as an input, and you use the typical BoW representation, you will need as many elements as there are words in the vocabulary. And that's just for representing one word. What if you wanted several neighboring words for context? With a language of 100k-1M vocabulary, you could easily blow through a million neurons. Look at the input of word2vec. $\endgroup$ – Emre Dec 28 '15 at 18:56
  • $\begingroup$ The convolutional neural networks (CNNs) that Martin mentioned are designed specifically with this issue in mind. The use of layers of partially connected receptive regions across many layers acts as a sort of efficient information filter that allows you to process millions of pixels without using millions of neurons. The algorithms is based on the means by which humans and cats use few optic pathway neurons to process huge amount of visual information and leads to the same sort of cost savings. $\endgroup$ – SQLServerSteve Aug 8 '16 at 17:29
6
$\begingroup$

There are several ways to make this big number trainable:

Dimensionality reduction of the input

  • Scale the image down
  • PCA / LDA

Troll-Answer

If you really meant "only a few neurons" then you might want to have a look at Spiking neural networks. Those are incredibly computationally intensive, need a lot of hand-crafting and still get worse performance than normal neural networks for most tasks ... but you only need very little of them.

$\endgroup$
2
$\begingroup$

Convolutional neural nets share weights across the whole input image and this drastically reduces the number of weights. For example you could have a 3x3 grid where each point has its own weight, this scans over each of the 3 channels in the 1024x1024 image. Only 9 weights are used for each channel making 27 weights. If you have, say 10 of these grids then that's only 270 weights!

Furthermore by sharing weights you create some translation invariance over the input space which is a desirable property in things like object recognition.

$\endgroup$
  • $\begingroup$ Thank you for your answer. Could you please name this method of image processing? I would like to read about it more detailed. $\endgroup$ – Iter Ator Dec 31 '15 at 16:56
  • $\begingroup$ cs231n.github.io/convolutional-networks is a good source for CNNs. $\endgroup$ – mattdns Dec 31 '15 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.