1
$\begingroup$

I was going through some lectures from the Deep Learning course that Geoffrey Hinton taught on Coursera and I came across this statement:

"RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects."

I have no idea what he meant nor seen/figured out any examples of this idea (and I'm sure other people are wondering too.) So if somebody would care to demonstrate, it would be much appreciated.

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't have the context, but I found a longer version of the quote from Yoshua Bengio that's more clear: “RNN could potentially learn to implement lots of small programs, using different subsets of its hidden states. And each of these little programs could capture nugget of knowledge and run in parallel, interacting to produce very complicated effects.” $\endgroup$
    – Emre
    May 11, 2017 at 4:56
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/220907/… $\endgroup$
    – Neil Slater
    May 11, 2017 at 6:56

1 Answer 1

Reset to default
1
$\begingroup$

Creating a model is having the computer write a small program. If you have a multi-layer network, then multiple nuggets might be interacting to produce complicated results.

That its pretty much how things like face detection work - you start with a series of really dumb "detectors" that look like basic rectangles and they combine in ways so that at the top, you're recognizing Brad Pitt.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is not what the quote is referring to. There is a key difference in behaviour of a RNN compared to a feed forward network. A feed forward NN model can approximate any function. A recurrent NN model can approximate any algorithm. $\endgroup$
    – Neil Slater
    Jun 10, 2017 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.