3
$\begingroup$

In a neural network, each neuron will have it's activation. But what the activation mean?

Does it just mean nothing but a temporal value to produce the final result or it's has something to with our understanding to the problem?

For example, considering a neural network to recognize a handwritten character from a picture, I wonder if it's possible that a neuron's activation represents the how the pixel matches the specific value in a small part of the picture?

$\endgroup$
5
$\begingroup$

The activation of a neuron is mathematically nothing but a function of its input. Consider a neural network with one hidden layer and one input vector $\mathbf{x}$. The input $a_j$ of neuron $j$ can be written as:

$a_j = w_j^T \mathbf{x} + b_j$.

The activation of neuron $j$ is then a transformation $g: \mathbf{R} \rightarrow \mathbf{R}$ of the input. For exmaple, one can use the sigmoid activiation function

$sig(a_j) = \dfrac{1}{1 + \exp(-a_j)} $

The weights $w$ are chosen to minimize a loss function and not for the sake of their interpretation. Nevertheless, one can try to interpret the activation of a neuron as internal representation of the input. One of the nicest examples I've seen comes from the Rummelhart et al (1986), Figure 4. In that paper, two family trees were fed into a neural network. The family trees represented an Italian and an English speaking family comprising three generations. Among other things, when activating one name of the tree, the neuronal activities represented whether the person was from the English or Italian tree and which generation the person embodied.

$\endgroup$
  • $\begingroup$ I thought now that the neural network is imitating the human's brain, so may be the neuron has something to do with human's understanding. $\endgroup$ – Zieng Apr 7 '16 at 2:02
  • $\begingroup$ @Zieng Yes..it actually does somehow. Imagine ur gf gives u a chocolate always when she comes to ur home. Now u went to some place and u saw that chocolate. You will remember her. That's because The neurones that connected the chocolate to ur gf has been activated. It means a neuron activates when it gets enough reason to. The collection of these activation will result in some meaningful results. The reason to activate was the image of chocolate. The result is ur gf's image. When u come to the maths, the sigmoid results 1 if there was high probability of the input in the data. In the chocolate $\endgroup$ – Amanuel Negash Apr 7 '16 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.