1
$\begingroup$

I am reading the paper with the title "Classification of indoor actions through deep neural networks". And I came across this statement:

With this mechanism very complex functions can be learned combining these modules: the resulting networks are often very sensitive to minute details and insensitive to large irrelevant variations.

I have read a lot about deep neural network. However, I might have missed the logic behind why deep neural networks are insensitive to large irrelevant features.

$\endgroup$
1
$\begingroup$

They can handle large variations of inputs because the neurons have weights and those weights get optimized as part of learning a good model. So even though the model might take in a value that ranges from 0 to 100,000, if it isn't relevant to predicting the outcome, it will have a very small effect.

| improve this answer | |
$\endgroup$
  • $\begingroup$ How come it's very sensitive to minute details as well? So you're saying the weight takes care of the irrelevant variations in the inputs? Hmm, what is the model is already trained and it accepts a new input? Can you give a more intuitive explanation? $\endgroup$ – mc8 May 9 '17 at 17:27
  • 1
    $\begingroup$ The features that are important get a high weight and are therefore sensitive while the features that are not important get a low weight. If this is a multi-layer model, some of the "features" can really composites of other neurons so it gets more complex, but that's the basic idea. If the model is already trained and gets a new input that follows the same pattern as the training data, the weights are (supposed) to take care of that. $\endgroup$ – CalZ May 9 '17 at 17:42
  • 1
    $\begingroup$ Intuitive example: you have a trained model in your head for assessing danger very quickly. If someone comes at you, you rapidly assess what they have in their hands. Assume a simple world where people can only hold bananas and knives. Holding_Banana has a weight of .000001 while Holding_Knife has a weight of 1. Tomorrow when you're walking around, it is largely irrelevant whether someone comes at you with a banana because of the weight. $\endgroup$ – CalZ May 9 '17 at 17:44
  • $\begingroup$ Oh right, cause I wasn't thinking in terms of the features. I remembered that deep nets extracts high level features as you add more layers. So it doesn't matter what the input is directly, cause the output depends on the features right? $\endgroup$ – mc8 May 9 '17 at 17:55
  • 1
    $\begingroup$ Generally, yes. If your features have really uneven scales they could in theory cause problems. People typically normalize or rescale values to handle that, or you hope you account for it in training. $\endgroup$ – CalZ May 9 '17 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.