Is it possible to decode which neuron represent which feature and why does it represent it?

Question

In a neural network, Each neuron in the network represents some part of non-linear feature of the input. Ex: Like in mnist data, Consider the stem of number 9 is cut into multiple pieces and different part is represented by different neurons in the first hidden layer(Just an example from 3B1B neural networks video). My questions are:

What determines which neuron get to represent which part of the stem??

Is it possible that if we pass in the same input multiple times, each neuron can represent different part of the stem?? OR Is it that its all the magic of chain rule (i.e At the beginning, all neuron represent some trash feature and as updation of weights occur and then particular features have become synonymous to a particular neuron.) If so, How does this happen?

Thanks in advance

Answer 1

Short answer: NO.

Think about it, you could have 5 or 5000 neurons, and each neuron would represent a different thing in different models with different inputs.

One of the problems of neural networks is that it is quite difficult to understand what is going inside them, they are a bit of a black box. Other techniques such as decision trees are much more easy to understand or explain to non-technical people.

The closest thing you could do is visualize the hidden layers and that might give you some sort of insight.

Answer 2

First, as 3B1B points out, the neurons don't necessarily end up as such clean features as segments then loops then whole numbers.

What determines which neuron get to represent which part of the stem??

Is it possible that if we pass in the same input multiple times, each neuron can represent different part of the stem?? OR Is it that its all the magic of chain rule (i.e At the beginning, all neuron represent some trash feature and as updation of weights occur and then particular features have become synonymous to a particular neuron.) If so, How does this happen?

It's the initial assignment of weights to the neurons, followed by the gradient descent (involving substantial chain rule applications). The "trash features" that each neuron starts with are terrible predictors, but they are "close to" different useful predictors. When applying gradient descent, they are pushed toward their respective "closest" improved features.

Unless I'm unfamiliar with some additional randomness in the popular implementations, if you fixed the initial weights and the order/batching of the inputs, you'd end up with the same network in the end. But, if say the inputs were passed in a different order, it's possible (likely?) that the gradient descent would push weights around differently enough that the final features were distributed to different neurons, or even completely dissimilar from the first run's features.