Typically all the layers of an artificial neural network are trainable. But what are the hidden layers trying to learn?

  • $\begingroup$ Doesn't this question sound a bit too open-ended? It could probably benefit from a narrative with the reasoning "a single perceptron layer has a known approximation goal, so what are each of the middle layers attempting to approximate on their own?". $\endgroup$ – E_net4 the flagger Jul 22 '17 at 22:21
  • $\begingroup$ I see your point but It was a valid interview question from a top company that is hiring deep learning engineers $\endgroup$ – George Pligoropoulos Jul 23 '17 at 14:26
  • $\begingroup$ Surely a company may attempt to test the interviewed with open-ended questions just to see how much they know. That IMO does not make it more appropriate for a Stack Exchange site. Even when self-answered, questions should be focused and show some effort. $\endgroup$ – E_net4 the flagger Jul 23 '17 at 15:22
  • $\begingroup$ @E_net4 Thank you for this perspective. However looking at my question again it does not seem to be missing anything. It seems quite to the point. My fear is that people in StackExchange, in general, have overfitted to the attributes that make a question good. This usually consists of medium length body (not small, not large), having a link, having a quotation, having some code <-- This makes a question look good because most good questions do look like this. But is this to say that missing those attributes a question is bad? $\endgroup$ – George Pligoropoulos Jul 24 '17 at 9:38
  • $\begingroup$ The Stack Overflow community has gone as far as to say that questions should show a lot of effort. The community-wide decision of making fairly substantial questions that usually contain code are only a realization of the fact that we are optimizing for pearls, not sand. While Data Science SE is a different site, and not a large one yet, we can (and should!) still strive for quality questions. I'll abstain from replying henceforth. We can take it to Meta if need be. $\endgroup$ – E_net4 the flagger Jul 24 '17 at 9:45

Each hidden layer represents a nonlinear application of a function on the inputs of the previous layer.

The first layers perform a function on our data-inputs and the last layer is responsible for either using softmax for classification purposes or be an affine layer for regression.

The in-between layers are necessary in order to stack complexity and therefore avoid having to create a layer of very high complexity.

In fact, in theory, two level deep neural network can represent any function. But how complex would this function need to be?
A layer of a very high complexity would be able to map our raw inputs into our final classification or regression task.

However what would be simpler to do is to let the intermediate hidden layers learn useful features regarding the given task and then the next hidden layer could exploit these features in its own way. And then the next hidden layer could use these newly transformed features in its own way and so on. So stacking nonlinear functions one on top of the other is more clever than requiring a single nonlinear function to do all the hard work.

There is a limit on how many layers you can stack together for the main reason that these hidden layers get trained by backpropagation which uses the chain rule of derivation in order to pass the deltas of the error back to the weights of the layers and tune them but a long series of these operations could end-up with exploding or vanishing gradients which means really poor performance or a model that cannot be trained.


Hidden layers try to find some kind of structure in the data. If you are working with images, visualizing hidden layers can show you what they are trying to learn. Here is an example: https://qph.ec.quoracdn.net/main-qimg-85bdd2809fb1d58c65262c909d22df59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.