1
$\begingroup$

I was going through some lectures from the Deep Learning course that Geoffrey Hinton taught on Coursera and I came across this statement:

"RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects."

I have no idea what he meant nor seen/figured out any examples of this idea (and I'm sure other people are wondering too.) So if somebody would care to demonstrate, it would be much appreciated.

$\endgroup$
  • $\begingroup$ I don't have the context, but I found a longer version of the quote from Yoshua Bengio that's more clear: “RNN could potentially learn to implement lots of small programs, using different subsets of its hidden states. And each of these little programs could capture nugget of knowledge and run in parallel, interacting to produce very complicated effects.” $\endgroup$ – Emre May 11 '17 at 4:56
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/220907/… $\endgroup$ – Neil Slater May 11 '17 at 6:56
1
$\begingroup$

Creating a model is having the computer write a small program. If you have a multi-layer network, then multiple nuggets might be interacting to produce complicated results.

That its pretty much how things like face detection work - you start with a series of really dumb "detectors" that look like basic rectangles and they combine in ways so that at the top, you're recognizing Brad Pitt.

| improve this answer | |
$\endgroup$
  • $\begingroup$ This is not what the quote is referring to. There is a key difference in behaviour of a RNN compared to a feed forward network. A feed forward NN model can approximate any function. A recurrent NN model can approximate any algorithm. $\endgroup$ – Neil Slater Jun 10 '17 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.