I'm running some experiments with NNs (actually I'm running an LSTM classifier), and I stumbled across a question I haven't found the answer so far.

What do NNs learn first? When we train a network for classifying objects, for example, the accuracy usually starts low on the training set and increases throughout the epochs. My question is, in these early epochs, which examples of my training set the network is correctly classifying? Are they the "easiest" examples or simply the examples that come first? It seems to me that they are the easiest examples, i.e., those that are easily distinguishable from each other.

I'd be happy if there's a mathematical explanation for that. Any help is appreciated!

  • $\begingroup$ This video might help. You may watch further videos in the same series $\endgroup$ – Kalsi Oct 3 '18 at 17:29

Potentially your looking for a description of the “vanishing or exploding gradient problem” in neural networks.

This post does a deep dive into this problem. The author states the issue briefly as:

an important observation: in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers.”

On a high level, we can look at the output of the neural network as a composition of functions $\hat{y} = f(g(h(x)))$.

After obtaining the loss, $y - \hat{y}$, we use gradient descent to begin to updating the weights. The weights in the deepest layer depend only on the derivative of the deepest function in the earlier composition, while shallow layers’ weights depend on the outer functions in the composition.

The authors explain that this results in a derivative for the shallow layers that includes the multiplication of the deeper weights, which are usually < 1 (in the vanishing gradient problem). Multiplying many numbers < 1 results in a very small number and a vanishing update to the weights in the shallow layers of the network.

| improve this answer | |
  • $\begingroup$ Thank you for your answer, but I think this is not really what I was looking for. I am aware of the vanishing gradient problem, and so deepest layers in the network learn faster than the early ones. My question is more on the interpretation of a general classification task with an NN. Which will be the first examples from my training set that the NN correctly predicts? $\endgroup$ – Lucas Dec 17 '18 at 20:47

Most likely it is learning whether or not the seeded connection weights are good or not. If you seed with different starting parameters the first few iterations will classify differently.

| improve this answer | |
  • $\begingroup$ Yes, I think that happens for the first batches. But I'm referring to something like after 3-5 epochs, for example. I think that the hits I get after these few epochs will likely be for the same classes (for a given dataset), regardless of the random initialization of parameters. Does that make sense? $\endgroup$ – Lucas Oct 10 '18 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.