What do neural networks learn first?

Question

I'm running some experiments with NNs (actually I'm running an LSTM classifier), and I stumbled across a question I haven't found the answer so far.

What do NNs learn first? When we train a network for classifying objects, for example, the accuracy usually starts low on the training set and increases throughout the epochs. My question is, in these early epochs, which examples of my training set the network is correctly classifying? Are they the "easiest" examples or simply the examples that come first? It seems to me that they are the easiest examples, i.e., those that are easily distinguishable from each other.

I'd be happy if there's a mathematical explanation for that. Any help is appreciated!

Answer 1

Potentially your looking for a description of the “vanishing or exploding gradient problem” in neural networks.

This post does a deep dive into this problem. The author states the issue briefly as:

an important observation: in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers.”

On a high level, we can look at the output of the neural network as a composition of functions $\hat{y} = f(g(h(x)))$.

After obtaining the loss, $y - \hat{y}$, we use gradient descent to begin to updating the weights. The weights in the deepest layer depend only on the derivative of the deepest function in the earlier composition, while shallow layers’ weights depend on the outer functions in the composition.

The authors explain that this results in a derivative for the shallow layers that includes the multiplication of the deeper weights, which are usually < 1 (in the vanishing gradient problem). Multiplying many numbers < 1 results in a very small number and a vanishing update to the weights in the shallow layers of the network.

Answer 2

Most likely it is learning whether or not the seeded connection weights are good or not. If you seed with different starting parameters the first few iterations will classify differently.